2019-10-25T14:19:58.796 INFO:root:teuthology version: 1.0.0-5052ff0 2019-10-25T14:19:58.797 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.fjRfRA.tmp -- --verbose --lock --description rados/singleton/{all/mon-config-key-caps.yaml msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/bluestore-avl.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} --name kchai-2019-10-25_07:20:39-rados-wip-kefu-testing-2019-10-25-0853-distro-basic-smithi --block --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2019-10-25_07:20:39-rados-wip-kefu-testing-2019-10-25-0853-distro-basic-smithi/4444511 2019-10-25T14:19:58.856 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-25T14:19:58.910 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2019-10-25_07:20:39-rados-wip-kefu-testing-2019-10-25-0853-distro-basic-smithi/4444511 branch: wip-kefu-testing-2019-10-25-0853 description: rados/singleton/{all/mon-config-key-caps.yaml msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/bluestore-avl.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} email: tchaikov@gmail.com first_in_suite: false job_id: '4444511' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2019-10-25_07:20:39-rados-wip-kefu-testing-2019-10-25-0853-distro-basic-smithi nuke-on-error: true os_type: centos os_version: '7.6' overrides: admin_socket: branch: wip-kefu-testing-2019-10-25-0853 ceph: conf: global: mon client directed command retry: 5 mon client hunt interval max multiple: 2 mon mgr beacon grace: 90 ms bind msgr2: false ms inject socket failures: 1000 ms type: async mgr: debug monc: 10 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: avl bluestore allocator: avl bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME mon_bind_msgr2: false sha1: 6eeafbc6a10ace643d1027657e8db6ca7ab0d58b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 6eeafbc6a10ace643d1027657e8db6ca7ab0d58b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-kefu-testing-2019-10-25-0853 sha1: 6eeafbc6a10ace643d1027657e8db6ca7ab0d58b owner: scheduled_kchai@teuthology priority: 99 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 6eeafbc6a10ace643d1027657e8db6ca7ab0d58b suite: rados suite_branch: wip-kefu-testing-2019-10-25-0853 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-10-25-0853/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 6eeafbc6a10ace643d1027657e8db6ca7ab0d58b tasks: - install: null - ceph: log-whitelist: - overall HEALTH_ - \(AUTH_BAD_CAPS\) - workunit: clients: all: - mon/test_config_key_caps.sh teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.23626 2019-10-25T14:19:58.910 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-10-25-0853/qa; will attempt to use it 2019-10-25T14:19:58.911 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-10-25-0853/qa/tasks 2019-10-25T14:19:58.911 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-10-25T14:19:58.913 INFO:teuthology.task.internal:Checking packages... 2019-10-25T14:19:58.930 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '6eeafbc6a10ace643d1027657e8db6ca7ab0d58b' 2019-10-25T14:19:58.930 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-10-25T14:19:58.930 INFO:teuthology.packaging:ref: None 2019-10-25T14:19:58.930 INFO:teuthology.packaging:tag: None 2019-10-25T14:19:58.931 INFO:teuthology.packaging:branch: wip-kefu-testing-2019-10-25-0853 2019-10-25T14:19:58.931 INFO:teuthology.packaging:sha1: 6eeafbc6a10ace643d1027657e8db6ca7ab0d58b 2019-10-25T14:19:58.931 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-kefu-testing-2019-10-25-0853 2019-10-25T14:19:59.115 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-6460.g6eeafbc 2019-10-25T14:19:59.115 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-10-25T14:19:59.121 INFO:teuthology.task.internal:no buildpackages task found 2019-10-25T14:19:59.121 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-10-25T14:19:59.167 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-10-25T14:19:59.167 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-25T14:19:59.210 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-10-25T14:20:09.227 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-10-25T14:20:19.241 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_kchai@teuthology', 'description': '/home/teuthworker/archive/kchai-2019-10-25_07:20:39-rados-wip-kefu-testing-2019-10-25-0853-distro-basic-smithi/4444511', 'machine_type': 'smithi'} 2019-10-25T14:20:19.256 DEBUG:teuthology.lock.ops:locked ubuntu@smithi166.front.sepia.ceph.com 2019-10-25T14:20:19.316 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-10-25T14:20:19.527 INFO:teuthology.provision.fog.smithi166:Scheduling deploy of centos 7.6 2019-10-25T14:20:19.884 INFO:teuthology.orchestra.console:Power off smithi166 2019-10-25T14:20:19.885 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-25T14:20:19.924 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-25T14:20:23.964 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-25T14:20:28.093 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-25T14:20:28.329 INFO:teuthology.orchestra.console:Power off for smithi166 completed 2019-10-25T14:20:28.431 INFO:teuthology.orchestra.console:Power on smithi166 2019-10-25T14:20:28.431 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-25T14:20:28.476 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-25T14:20:32.516 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-25T14:20:36.645 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-25T14:20:36.875 INFO:teuthology.orchestra.console:Power on for smithi166 completed 2019-10-25T14:20:36.976 INFO:teuthology.provision.fog.smithi166:Waiting for deploy to finish 2019-10-25T14:23:08.082 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi166.front.sepia.ceph.com', 'timeout': 60} 2019-10-25T14:23:17.169 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi166.front.sepia.ceph.com', 'timeout': 60} 2019-10-25T14:23:26.253 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi166.front.sepia.ceph.com', 'timeout': 60} 2019-10-25T14:23:35.313 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi166.front.sepia.ceph.com', 'timeout': 60} 2019-10-25T14:23:44.398 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi166.front.sepia.ceph.com', 'timeout': 60} 2019-10-25T14:23:53.458 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi166.front.sepia.ceph.com', 'timeout': 60} 2019-10-25T14:24:02.540 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi166.front.sepia.ceph.com', 'timeout': 60} 2019-10-25T14:24:34.257 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:34.257 INFO:teuthology.orchestra.run.smithi166:> hostname 2019-10-25T14:24:34.876 INFO:teuthology.orchestra.run.smithi166.stdout:smithi141 2019-10-25T14:24:34.876 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:34.877 INFO:teuthology.orchestra.run.smithi166:> grep smithi141 /etc/hosts 2019-10-25T14:24:34.937 INFO:teuthology.orchestra.run.smithi166.stdout:172.21.15.141 smithi141.front.sepia.ceph.com smithi141 2019-10-25T14:24:34.937 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:34.938 INFO:teuthology.orchestra.run.smithi166:> sudo hostname smithi166 2019-10-25T14:24:35.075 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:35.075 INFO:teuthology.orchestra.run.smithi166:> sudo sed -i -e 's/smithi141/smithi166/g' /etc/hosts 2019-10-25T14:24:35.158 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:35.158 INFO:teuthology.orchestra.run.smithi166:> sudo sed -i -e 's/172.21.15.141/172.21.15.166/g' /etc/hosts 2019-10-25T14:24:35.215 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:35.215 INFO:teuthology.orchestra.run.smithi166:> sudo sed -i -e 's/smithi141/smithi166/g' /etc/hostname 2019-10-25T14:24:35.272 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:35.273 INFO:teuthology.orchestra.run.smithi166:> sudo hostname smithi166 2019-10-25T14:24:35.329 INFO:teuthology.provision.fog.smithi166:Deploy complete! 2019-10-25T14:24:35.331 DEBUG:teuthology.parallel:result is None 2019-10-25T14:24:35.469 INFO:teuthology.lock.keys:Checking smithi166.front.sepia.ceph.com 2019-10-25T14:24:35.470 INFO:teuthology.lock.keys:New key found. Updating... 2019-10-25T14:24:35.488 INFO:teuthology.lock.ops:Updating [smithi166.front.sepia.ceph.com]: set os type and version on server 2019-10-25T14:24:35.488 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-25T14:24:35.489 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi166.front.sepia.ceph.com', 'timeout': 60} 2019-10-25T14:24:35.575 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:35.576 INFO:teuthology.orchestra.run.smithi166:> true 2019-10-25T14:24:35.854 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:35.854 INFO:teuthology.orchestra.run.smithi166:> uname -m 2019-10-25T14:24:35.901 INFO:teuthology.orchestra.run.smithi166.stdout:x86_64 2019-10-25T14:24:35.902 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:35.902 INFO:teuthology.orchestra.run.smithi166:> cat /etc/os-release 2019-10-25T14:24:35.944 INFO:teuthology.orchestra.run.smithi166.stdout:NAME="CentOS Linux" 2019-10-25T14:24:35.944 INFO:teuthology.orchestra.run.smithi166.stdout:VERSION="7 (Core)" 2019-10-25T14:24:35.945 INFO:teuthology.orchestra.run.smithi166.stdout:ID="centos" 2019-10-25T14:24:35.945 INFO:teuthology.orchestra.run.smithi166.stdout:ID_LIKE="rhel fedora" 2019-10-25T14:24:35.945 INFO:teuthology.orchestra.run.smithi166.stdout:VERSION_ID="7" 2019-10-25T14:24:35.945 INFO:teuthology.orchestra.run.smithi166.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-10-25T14:24:35.945 INFO:teuthology.orchestra.run.smithi166.stdout:ANSI_COLOR="0;31" 2019-10-25T14:24:35.945 INFO:teuthology.orchestra.run.smithi166.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-10-25T14:24:35.945 INFO:teuthology.orchestra.run.smithi166.stdout:HOME_URL="https://www.centos.org/" 2019-10-25T14:24:35.946 INFO:teuthology.orchestra.run.smithi166.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-10-25T14:24:35.946 INFO:teuthology.orchestra.run.smithi166.stdout: 2019-10-25T14:24:35.946 INFO:teuthology.orchestra.run.smithi166.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-10-25T14:24:35.946 INFO:teuthology.orchestra.run.smithi166.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-10-25T14:24:35.946 INFO:teuthology.orchestra.run.smithi166.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-10-25T14:24:35.946 INFO:teuthology.orchestra.run.smithi166.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-10-25T14:24:35.946 INFO:teuthology.orchestra.run.smithi166.stdout: 2019-10-25T14:24:35.948 INFO:teuthology.lock.ops:Updating smithi166.front.sepia.ceph.com on lock server 2019-10-25T14:24:35.961 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2019-10-25T14:24:35.975 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi166.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnhk5B4Crg9zY5jGaqPH7ZmK+RlatNcV20sUVFWqOgAUvaYQnn5moJJrmx3iw/P1/k7yneuF23neW66HLDCcVk0bSGjweL133hF5pUzhs5OdmBPqQigAV4dgRm93MoNes52LAxRRLqplLKbgHcPMDhCADNjwI/5gT65SGimhpqwGugU13ETlWgmZvKQuPeuV39Da4zXsQRM6mBPUIOiDW9Omk8TgR7BLwkPV66iMnUDzhbMT9AdvJqZN3gTlRIQDYx3oE+19avu8lvK1JW1jGpe6XWuMR5GjtN8DwqJdjBeOA8wC7EtQ/7AipyJsZuTCQ/jlYZwkVEIhXaGNab4PsR 2019-10-25T14:24:35.975 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-25T14:24:36.013 INFO:teuthology.run_tasks:Running task internal.save_config... 2019-10-25T14:24:36.022 INFO:teuthology.task.internal:Saving configuration 2019-10-25T14:24:36.042 INFO:teuthology.run_tasks:Running task internal.check_lock... 2019-10-25T14:24:36.058 INFO:teuthology.task.internal.check_lock:Checking locks... 2019-10-25T14:24:36.071 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-10-25 14:20:19.248750', u'locked_by': u'scheduled_kchai@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi166.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNQH7GSz4xV8FV1KHcJqGM67K0nSzDFECVggp8xpJmX5GzawPhzaisdzpurRyFDy1g/N+kMDS4nZEui0cU4MfNE=', u'description': u'/home/teuthworker/archive/kchai-2019-10-25_07:20:39-rados-wip-kefu-testing-2019-10-25-0853-distro-basic-smithi/4444511'} 2019-10-25T14:24:36.072 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2019-10-25T14:24:36.083 INFO:teuthology.task.internal:roles: ubuntu@smithi166.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2019-10-25T14:24:36.083 INFO:teuthology.run_tasks:Running task console_log... 2019-10-25T14:24:36.160 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-10-25T14:24:36.161 INFO:teuthology.run_tasks:Running task internal.connect... 2019-10-25T14:24:36.171 INFO:teuthology.task.internal:Opening connections... 2019-10-25T14:24:36.172 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi166.front.sepia.ceph.com 2019-10-25T14:24:36.173 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi166.front.sepia.ceph.com', 'timeout': 60} 2019-10-25T14:24:36.277 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2019-10-25T14:24:36.307 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:36.307 INFO:teuthology.orchestra.run.smithi166:> uname -m 2019-10-25T14:24:36.553 INFO:teuthology.orchestra.run.smithi166.stdout:x86_64 2019-10-25T14:24:36.554 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:36.554 INFO:teuthology.orchestra.run.smithi166:> cat /etc/os-release 2019-10-25T14:24:36.611 INFO:teuthology.orchestra.run.smithi166.stdout:NAME="CentOS Linux" 2019-10-25T14:24:36.612 INFO:teuthology.orchestra.run.smithi166.stdout:VERSION="7 (Core)" 2019-10-25T14:24:36.612 INFO:teuthology.orchestra.run.smithi166.stdout:ID="centos" 2019-10-25T14:24:36.612 INFO:teuthology.orchestra.run.smithi166.stdout:ID_LIKE="rhel fedora" 2019-10-25T14:24:36.612 INFO:teuthology.orchestra.run.smithi166.stdout:VERSION_ID="7" 2019-10-25T14:24:36.613 INFO:teuthology.orchestra.run.smithi166.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-10-25T14:24:36.613 INFO:teuthology.orchestra.run.smithi166.stdout:ANSI_COLOR="0;31" 2019-10-25T14:24:36.613 INFO:teuthology.orchestra.run.smithi166.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-10-25T14:24:36.613 INFO:teuthology.orchestra.run.smithi166.stdout:HOME_URL="https://www.centos.org/" 2019-10-25T14:24:36.613 INFO:teuthology.orchestra.run.smithi166.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-10-25T14:24:36.613 INFO:teuthology.orchestra.run.smithi166.stdout: 2019-10-25T14:24:36.613 INFO:teuthology.orchestra.run.smithi166.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-10-25T14:24:36.614 INFO:teuthology.orchestra.run.smithi166.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-10-25T14:24:36.615 INFO:teuthology.orchestra.run.smithi166.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-10-25T14:24:36.615 INFO:teuthology.orchestra.run.smithi166.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-10-25T14:24:36.615 INFO:teuthology.orchestra.run.smithi166.stdout: 2019-10-25T14:24:36.616 INFO:teuthology.lock.ops:Updating smithi166.front.sepia.ceph.com on lock server 2019-10-25T14:24:36.637 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2019-10-25T14:24:36.870 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2019-10-25T14:24:36.973 INFO:teuthology.task.internal:Checking for old test directory... 2019-10-25T14:24:36.973 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:36.973 INFO:teuthology.orchestra.run.smithi166:> test '!' -e /home/ubuntu/cephtest 2019-10-25T14:24:37.016 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2019-10-25T14:24:37.084 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2019-10-25T14:24:37.084 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:37.084 INFO:teuthology.orchestra.run.smithi166:> test -z $(ls -A /var/lib/ceph) 2019-10-25T14:24:37.131 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2019-10-25T14:24:37.225 INFO:teuthology.run_tasks:Running task kernel... 2019-10-25T14:24:37.241 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2019-10-25T14:24:37.242 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2019-10-25T14:24:37.242 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@smithi166.front.sepia.ceph.com 2019-10-25T14:24:37.242 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:37.242 INFO:teuthology.orchestra.run.smithi166:> sudo package-cleanup -y --oldkernels 2019-10-25T14:24:38.013 INFO:teuthology.orchestra.run.smithi166.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-25T14:24:41.741 INFO:teuthology.orchestra.run.smithi166.stdout:No old kernels to remove 2019-10-25T14:24:41.754 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:41.754 INFO:teuthology.orchestra.run.smithi166:> uname -r 2019-10-25T14:24:41.810 INFO:teuthology.orchestra.run.smithi166.stdout:3.10.0-957.27.2.el7.x86_64 2019-10-25T14:24:41.811 INFO:teuthology.task.kernel:Running kernel on smithi166: 3.10.0-957.27.2.el7.x86_64 2019-10-25T14:24:41.811 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:24:41.811 INFO:teuthology.orchestra.run.smithi166:> sudo yum install -y kernel 2019-10-25T14:24:42.037 INFO:teuthology.orchestra.run.smithi166.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-25T14:24:42.095 INFO:teuthology.orchestra.run.smithi166.stdout:Determining fastest mirrors 2019-10-25T14:24:45.184 INFO:teuthology.orchestra.run.smithi166.stdout: * base: packages.oit.ncsu.edu 2019-10-25T14:24:45.184 INFO:teuthology.orchestra.run.smithi166.stdout: * epel: ftp.linux.ncsu.edu 2019-10-25T14:24:45.185 INFO:teuthology.orchestra.run.smithi166.stdout: * extras: mirror.fileplanet.com 2019-10-25T14:24:45.187 INFO:teuthology.orchestra.run.smithi166.stdout: * updates: mirror.cs.vt.edu 2019-10-25T14:24:48.664 INFO:teuthology.orchestra.run.smithi166.stdout:1 packages excluded due to repository priority protections 2019-10-25T14:24:49.032 INFO:teuthology.orchestra.run.smithi166.stdout:Resolving Dependencies 2019-10-25T14:24:49.033 INFO:teuthology.orchestra.run.smithi166.stdout:--> Running transaction check 2019-10-25T14:24:49.033 INFO:teuthology.orchestra.run.smithi166.stdout:---> Package kernel.x86_64 0:3.10.0-1062.4.1.el7 will be installed 2019-10-25T14:24:49.281 INFO:teuthology.orchestra.run.smithi166.stdout:--> Processing Dependency: linux-firmware >= 20190429-72 for package: kernel-3.10.0-1062.4.1.el7.x86_64 2019-10-25T14:24:49.390 INFO:teuthology.orchestra.run.smithi166.stdout:--> Running transaction check 2019-10-25T14:24:49.390 INFO:teuthology.orchestra.run.smithi166.stdout:---> Package linux-firmware.noarch 0:20180911-69.git85c5d90.el7 will be updated 2019-10-25T14:24:49.428 INFO:teuthology.orchestra.run.smithi166.stdout:---> Package linux-firmware.noarch 0:20190429-72.gitddde598.el7 will be an update 2019-10-25T14:24:50.567 INFO:teuthology.orchestra.run.smithi166.stdout:--> Finished Dependency Resolution 2019-10-25T14:24:51.204 INFO:teuthology.orchestra.run.smithi166.stdout: 2019-10-25T14:24:51.205 INFO:teuthology.orchestra.run.smithi166.stdout:Dependencies Resolved 2019-10-25T14:24:51.218 INFO:teuthology.orchestra.run.smithi166.stdout: 2019-10-25T14:24:51.219 INFO:teuthology.orchestra.run.smithi166.stdout:================================================================================ 2019-10-25T14:24:51.219 INFO:teuthology.orchestra.run.smithi166.stdout: Package Arch Version Repository Size 2019-10-25T14:24:51.219 INFO:teuthology.orchestra.run.smithi166.stdout:================================================================================ 2019-10-25T14:24:51.220 INFO:teuthology.orchestra.run.smithi166.stdout:Installing: 2019-10-25T14:24:51.220 INFO:teuthology.orchestra.run.smithi166.stdout: kernel x86_64 3.10.0-1062.4.1.el7 updates 50 M 2019-10-25T14:24:51.220 INFO:teuthology.orchestra.run.smithi166.stdout:Updating for dependencies: 2019-10-25T14:24:51.220 INFO:teuthology.orchestra.run.smithi166.stdout: linux-firmware noarch 20190429-72.gitddde598.el7 base 73 M 2019-10-25T14:24:51.220 INFO:teuthology.orchestra.run.smithi166.stdout: 2019-10-25T14:24:51.221 INFO:teuthology.orchestra.run.smithi166.stdout:Transaction Summary 2019-10-25T14:24:51.221 INFO:teuthology.orchestra.run.smithi166.stdout:================================================================================ 2019-10-25T14:24:51.221 INFO:teuthology.orchestra.run.smithi166.stdout:Install 1 Package 2019-10-25T14:24:51.221 INFO:teuthology.orchestra.run.smithi166.stdout:Upgrade ( 1 Dependent package) 2019-10-25T14:24:51.221 INFO:teuthology.orchestra.run.smithi166.stdout: 2019-10-25T14:24:51.221 INFO:teuthology.orchestra.run.smithi166.stdout:Total download size: 123 M 2019-10-25T14:24:51.222 INFO:teuthology.orchestra.run.smithi166.stdout:Downloading packages: 2019-10-25T14:24:51.222 INFO:teuthology.orchestra.run.smithi166.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-10-25T14:24:52.358 INFO:teuthology.orchestra.run.smithi166.stdout:-------------------------------------------------------------------------------- 2019-10-25T14:24:52.358 INFO:teuthology.orchestra.run.smithi166.stdout:Total 108 MB/s | 123 MB 00:01 2019-10-25T14:24:52.430 INFO:teuthology.orchestra.run.smithi166.stdout:Running transaction check 2019-10-25T14:24:52.603 INFO:teuthology.orchestra.run.smithi166.stdout:Running transaction test 2019-10-25T14:24:53.655 INFO:teuthology.orchestra.run.smithi166.stdout:Transaction test succeeded 2019-10-25T14:24:53.656 INFO:teuthology.orchestra.run.smithi166.stdout:Running transaction 2019-10-25T14:25:04.095 INFO:teuthology.orchestra.run.smithi166.stdout: Updating : linux-firmware-20190429-72.gitddde598.el7.noarch 1/3 2019-10-25T14:25:08.876 INFO:teuthology.orchestra.run.smithi166.stdout: Installing : kernel-3.10.0-1062.4.1.el7.x86_64 2/3 2019-10-25T14:25:35.328 INFO:teuthology.orchestra.run.smithi166.stdout: Cleanup : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2019-10-25T14:25:35.477 INFO:teuthology.orchestra.run.smithi166.stdout: Verifying : kernel-3.10.0-1062.4.1.el7.x86_64 1/3 2019-10-25T14:25:35.593 INFO:teuthology.orchestra.run.smithi166.stdout: Verifying : linux-firmware-20190429-72.gitddde598.el7.noarch 2/3 2019-10-25T14:25:35.941 INFO:teuthology.orchestra.run.smithi166.stdout: Verifying : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2019-10-25T14:25:35.941 INFO:teuthology.orchestra.run.smithi166.stdout: 2019-10-25T14:25:35.941 INFO:teuthology.orchestra.run.smithi166.stdout:Installed: 2019-10-25T14:25:35.941 INFO:teuthology.orchestra.run.smithi166.stdout: kernel.x86_64 0:3.10.0-1062.4.1.el7 2019-10-25T14:25:35.941 INFO:teuthology.orchestra.run.smithi166.stdout: 2019-10-25T14:25:35.942 INFO:teuthology.orchestra.run.smithi166.stdout:Dependency Updated: 2019-10-25T14:25:35.942 INFO:teuthology.orchestra.run.smithi166.stdout: linux-firmware.noarch 0:20190429-72.gitddde598.el7 2019-10-25T14:25:35.942 INFO:teuthology.orchestra.run.smithi166.stdout: 2019-10-25T14:25:35.942 INFO:teuthology.orchestra.run.smithi166.stdout:Complete! 2019-10-25T14:25:36.073 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:25:36.073 INFO:teuthology.orchestra.run.smithi166:> rpm -q kernel --last | head -n 1 2019-10-25T14:25:36.541 INFO:teuthology.orchestra.run.smithi166.stdout:kernel-3.10.0-1062.4.1.el7.x86_64 Fri 25 Oct 2019 02:25:04 PM UTC 2019-10-25T14:25:36.545 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.4.1.el7.x86_64 2019-10-25T14:25:36.545 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-957.27.2.el7.x86_64 Expected: 3.10.0-1062.4.1.el7.x86_64 2019-10-25T14:25:36.545 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2019-10-25T14:25:36.545 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-10-25T14:25:36.545 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2019-10-25T14:25:36.546 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi166.front.sepia.ceph.com, path=None, version=distro) 2019-10-25T14:25:36.546 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:25:36.546 INFO:teuthology.orchestra.run.smithi166:> rpm -q kernel --last | head -n 1 2019-10-25T14:25:36.802 INFO:teuthology.orchestra.run.smithi166.stdout:kernel-3.10.0-1062.4.1.el7.x86_64 Fri 25 Oct 2019 02:25:04 PM UTC 2019-10-25T14:25:36.804 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.4.1.el7.x86_64 2019-10-25T14:25:36.804 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:25:36.805 INFO:teuthology.orchestra.run.smithi166:> sudo rpm -qi grub2 2019-10-25T14:25:36.945 INFO:teuthology.orchestra.run.smithi166.stdout:Name : grub2 2019-10-25T14:25:36.945 INFO:teuthology.orchestra.run.smithi166.stdout:Epoch : 1 2019-10-25T14:25:36.945 INFO:teuthology.orchestra.run.smithi166.stdout:Version : 2.02 2019-10-25T14:25:36.945 INFO:teuthology.orchestra.run.smithi166.stdout:Release : 0.76.el7.centos 2019-10-25T14:25:36.945 INFO:teuthology.orchestra.run.smithi166.stdout:Architecture: x86_64 2019-10-25T14:25:36.945 INFO:teuthology.orchestra.run.smithi166.stdout:Install Date: Wed 14 Aug 2019 05:05:05 PM UTC 2019-10-25T14:25:36.946 INFO:teuthology.orchestra.run.smithi166.stdout:Group : System Environment/Base 2019-10-25T14:25:36.946 INFO:teuthology.orchestra.run.smithi166.stdout:Size : 0 2019-10-25T14:25:36.946 INFO:teuthology.orchestra.run.smithi166.stdout:License : GPLv3+ 2019-10-25T14:25:36.946 INFO:teuthology.orchestra.run.smithi166.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:06 PM UTC, Key ID 24c6a8a7f4a80eb5 2019-10-25T14:25:36.946 INFO:teuthology.orchestra.run.smithi166.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2019-10-25T14:25:36.946 INFO:teuthology.orchestra.run.smithi166.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2019-10-25T14:25:36.946 INFO:teuthology.orchestra.run.smithi166.stdout:Build Host : kbuilder.bsys.centos.org 2019-10-25T14:25:36.947 INFO:teuthology.orchestra.run.smithi166.stdout:Relocations : (not relocatable) 2019-10-25T14:25:36.947 INFO:teuthology.orchestra.run.smithi166.stdout:Packager : CentOS BuildSystem 2019-10-25T14:25:36.947 INFO:teuthology.orchestra.run.smithi166.stdout:Vendor : CentOS 2019-10-25T14:25:36.947 INFO:teuthology.orchestra.run.smithi166.stdout:URL : http://www.gnu.org/software/grub/ 2019-10-25T14:25:36.947 INFO:teuthology.orchestra.run.smithi166.stdout:Summary : Bootloader with support for Linux, Multiboot and more 2019-10-25T14:25:36.947 INFO:teuthology.orchestra.run.smithi166.stdout:Description : 2019-10-25T14:25:36.947 INFO:teuthology.orchestra.run.smithi166.stdout: 2019-10-25T14:25:36.947 INFO:teuthology.orchestra.run.smithi166.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2019-10-25T14:25:36.948 INFO:teuthology.orchestra.run.smithi166.stdout:customizable bootloader with modular architecture. It supports a rich 2019-10-25T14:25:36.948 INFO:teuthology.orchestra.run.smithi166.stdout:variety of kernel formats, file systems, computer architectures and 2019-10-25T14:25:36.948 INFO:teuthology.orchestra.run.smithi166.stdout:hardware devices. 2019-10-25T14:25:36.949 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2019-10-25T14:25:36.949 INFO:teuthology.task.kernel:Updating grub on smithi166 to boot 3.10.0-1062.4.1.el7.x86_64 2019-10-25T14:25:36.949 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:25:36.949 INFO:teuthology.orchestra.run.smithi166:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2019-10-25T14:25:37.403 INFO:teuthology.orchestra.run.smithi166.stderr:Generating grub configuration file ... 2019-10-25T14:25:37.576 INFO:teuthology.orchestra.run.smithi166.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.4.1.el7.x86_64 2019-10-25T14:25:37.600 INFO:teuthology.orchestra.run.smithi166.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.4.1.el7.x86_64.img 2019-10-25T14:25:37.855 INFO:teuthology.orchestra.run.smithi166.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.27.2.el7.x86_64 2019-10-25T14:25:37.882 INFO:teuthology.orchestra.run.smithi166.stderr:Found initrd image: /boot/initramfs-3.10.0-957.27.2.el7.x86_64.img 2019-10-25T14:25:37.917 INFO:teuthology.orchestra.run.smithi166.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2019-10-25T14:25:37.940 INFO:teuthology.orchestra.run.smithi166.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2019-10-25T14:25:37.969 INFO:teuthology.orchestra.run.smithi166.stderr:Found linux image: /boot/vmlinuz-0-rescue-48f603b7a56f4bb39f72becf12f3b725 2019-10-25T14:25:37.977 INFO:teuthology.orchestra.run.smithi166.stderr:Found initrd image: /boot/initramfs-0-rescue-48f603b7a56f4bb39f72becf12f3b725.img 2019-10-25T14:25:39.225 INFO:teuthology.orchestra.run.smithi166.stderr:done 2019-10-25T14:25:39.230 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:25:39.230 INFO:teuthology.orchestra.run.smithi166:> mktemp 2019-10-25T14:25:39.293 INFO:teuthology.orchestra.run.smithi166.stdout:/tmp/tmp.EmmCcGZCSo 2019-10-25T14:25:39.294 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:25:39.294 INFO:teuthology.orchestra.run.smithi166:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EmmCcGZCSo 2019-10-25T14:25:39.366 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:25:39.366 INFO:teuthology.orchestra.run.smithi166:> sudo chmod 0666 /tmp/tmp.EmmCcGZCSo 2019-10-25T14:25:39.499 DEBUG:teuthology.orchestra.remote:smithi166:/tmp/tmp.EmmCcGZCSo is 6KB 2019-10-25T14:25:39.556 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:25:39.557 INFO:teuthology.orchestra.run.smithi166:> rm -fr /tmp/tmp.EmmCcGZCSo 2019-10-25T14:25:39.604 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:25:39.604 INFO:teuthology.orchestra.run.smithi166:> sudo grub2-set-default 0 2019-10-25T14:25:39.865 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:25:39.865 INFO:teuthology.orchestra.run.smithi166:> sudo shutdown -r now 2019-10-25T14:25:39.876 INFO:teuthology.misc:Re-opening connections... 2019-10-25T14:25:39.877 INFO:teuthology.misc:trying to connect to ubuntu@smithi166.front.sepia.ceph.com 2019-10-25T14:25:39.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-25T14:25:39.879 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi166.front.sepia.ceph.com', 'timeout': 60} 2019-10-25T14:25:39.983 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:25:39.983 INFO:teuthology.orchestra.run.smithi166:> true 2019-10-25T14:25:40.303 DEBUG:teuthology.misc:waited 0.425936937332 2019-10-25T14:25:41.304 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2019-10-25T14:25:41.304 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "3.10.0-1062.4.1.el7.x86_64"... 2019-10-25T14:25:41.304 INFO:teuthology.orchestra.run.smithi166:Running: 2019-10-25T14:25:41.304 INFO:teuthology.orchestra.run.smithi166:> uname -r 2019-10-26T00:05:01.718 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2019-10-26T00:05:01.760 DEBUG:teuthology.task.console_log:Killing console logger for smithi166 2019-10-26T00:05:01.761 DEBUG:teuthology.task.console_log:Killing console logger for smithi166 2019-10-26T00:05:01.761 DEBUG:teuthology.exit:Finished running handlers