2019-10-11T12:48:03.500 INFO:root:teuthology version: 1.0.0-471e0e3 2019-10-11T12:48:03.501 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.6RsOH1.tmp -- --verbose --lock --description rados/perf/{ceph.yaml objectstore/bluestore-stupid.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/fio_4M_rand_read.yaml} --name kchai-2019-10-11_00:28:24-rados-wip-kefu-testing-2019-10-09-1243-distro-basic-mira --block --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2019-10-11_00:28:24-rados-wip-kefu-testing-2019-10-09-1243-distro-basic-mira/4400393 2019-10-11T12:48:03.550 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-11T12:48:03.606 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2019-10-11_00:28:24-rados-wip-kefu-testing-2019-10-09-1243-distro-basic-mira/4400393 branch: wip-kefu-testing-2019-10-09-1243 description: rados/perf/{ceph.yaml objectstore/bluestore-stupid.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/fio_4M_rand_read.yaml} email: null first_in_suite: false job_id: '4400393' kernel: kdb: true sha1: distro last_in_suite: false machine_type: mira name: kchai-2019-10-11_00:28:24-rados-wip-kefu-testing-2019-10-09-1243-distro-basic-mira nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: rhel os_version: '7.7' overrides: admin_socket: branch: wip-kefu-testing-2019-10-09-1243 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 0dca79308d83afd02b99aa1aa8839d156215298c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 0dca79308d83afd02b99aa1aa8839d156215298c thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-kefu-testing-2019-10-09-1243 sha1: 0dca79308d83afd02b99aa1aa8839d156215298c owner: scheduled_kchai@teuthology priority: 70 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 0dca79308d83afd02b99aa1aa8839d156215298c suite: rados suite_branch: wip-kefu-testing-2019-10-09-1243 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-10-09-1243/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 0dca79308d83afd02b99aa1aa8839d156215298c tasks: - install: null - ceph: fs: xfs log-whitelist: - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH wait-for-scrub: false - rgw: - client.0 - ssh_keys: null - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randread norandommap: true op_size: - 4194304 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu teuthology_branch: master tube: mira verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.mira.11872 2019-10-11T12:48:03.607 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-10-09-1243/qa; will attempt to use it 2019-10-11T12:48:03.607 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-10-09-1243/qa/tasks 2019-10-11T12:48:03.607 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-10-11T12:48:03.612 INFO:teuthology.task.internal:Checking packages... 2019-10-11T12:48:03.624 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash '0dca79308d83afd02b99aa1aa8839d156215298c' 2019-10-11T12:48:03.625 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-10-11T12:48:03.625 INFO:teuthology.packaging:ref: None 2019-10-11T12:48:03.625 INFO:teuthology.packaging:tag: None 2019-10-11T12:48:03.625 INFO:teuthology.packaging:branch: wip-kefu-testing-2019-10-09-1243 2019-10-11T12:48:03.625 INFO:teuthology.packaging:sha1: 0dca79308d83afd02b99aa1aa8839d156215298c 2019-10-11T12:48:03.625 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-kefu-testing-2019-10-09-1243 2019-10-11T12:48:03.793 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-6009.g0dca793 2019-10-11T12:48:03.793 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-10-11T12:48:03.992 INFO:teuthology.task.internal:no buildpackages task found 2019-10-11T12:48:03.992 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-10-11T12:48:04.006 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-10-11T12:48:04.006 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-11T12:48:04.067 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:48:14.084 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:48:24.100 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:48:34.111 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:48:44.125 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:48:54.141 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:49:04.149 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:49:14.160 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:49:24.176 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:49:34.193 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:49:44.204 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:49:54.220 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:50:04.236 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:50:14.252 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:50:24.269 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:50:34.287 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:50:44.300 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:50:54.318 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:51:04.332 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:51:14.349 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:51:24.367 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:51:34.384 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:51:44.400 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:51:54.409 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:52:04.425 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:52:14.443 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:52:24.455 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:52:34.468 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:52:44.485 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:52:54.502 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:53:04.519 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:53:14.532 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:53:24.548 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:53:34.561 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:53:44.575 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:53:54.592 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:54:04.611 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:54:14.624 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:54:24.642 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:54:34.656 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:54:44.672 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:54:54.688 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:55:04.706 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:55:14.719 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:55:24.736 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:55:34.752 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:55:44.767 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 1, have 2)... 2019-10-11T12:55:54.791 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_kchai@teuthology', 'description': '/home/teuthworker/archive/kchai-2019-10-11_00:28:24-rados-wip-kefu-testing-2019-10-09-1243-distro-basic-mira/4400393', 'machine_type': 'mira'} 2019-10-11T12:55:54.804 DEBUG:teuthology.lock.ops:locked ubuntu@mira101.front.sepia.ceph.com 2019-10-11T12:55:54.876 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-10-11T12:55:55.064 INFO:teuthology.provision.fog.mira101:Scheduling deploy of rhel 7.7 2019-10-11T12:55:55.384 INFO:teuthology.orchestra.console:Power off mira101 2019-10-11T12:55:55.385 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-11T12:55:55.438 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:55:55.583 INFO:teuthology.orchestra.console:Power off for mira101 completed 2019-10-11T12:55:55.684 INFO:teuthology.orchestra.console:Power on mira101 2019-10-11T12:55:55.684 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-11T12:55:55.729 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:55:59.776 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:56:03.913 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:56:04.141 INFO:teuthology.orchestra.console:Power on for mira101 completed 2019-10-11T12:56:04.244 INFO:teuthology.provision.fog.mira101:Waiting for deploy to finish 2019-10-11T12:58:36.049 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira101.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:58:45.138 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira101.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:58:54.217 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira101.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:03.277 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira101.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:12.333 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira101.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:21.385 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira101.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:29.052 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2019-10-11T12:59:29.052 DEBUG:teuthology.task.console_log:Killing console logger for mira101 2019-10-11T12:59:29.053 DEBUG:teuthology.exit:Finished running handlers