2019-09-28T23:37:06.923 INFO:root:teuthology version: 1.0.0-df92266 2019-09-28T23:37:06.924 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.7v_peI.tmp -- --verbose --lock --description rados/singleton-nomsgr/{all/cache-fs-trunc.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} --name sjust-2019-09-28_23:02:01-rados-wip-sjust-testing-distro-basic-smithi --block --owner scheduled_sjust@teuthology --archive /home/teuthworker/archive/sjust-2019-09-28_23:02:01-rados-wip-sjust-testing-distro-basic-smithi/4342933 2019-09-28T23:37:06.971 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-09-28T23:37:07.029 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sjust-2019-09-28_23:02:01-rados-wip-sjust-testing-distro-basic-smithi/4342933 branch: wip-sjust-testing description: rados/singleton-nomsgr/{all/cache-fs-trunc.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} email: null first_in_suite: false job_id: '4342933' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sjust-2019-09-28_23:02:01-rados-wip-sjust-testing-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '7.6' overrides: admin_socket: branch: wip-sjust-testing ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 osd debug verify cached snaps: true osd debug verify missing on start: true osd op queue: debug_random osd op queue cut off: debug_random log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: a2ea9c913f59e2c5615ea0313e506180d28b2f4e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: a2ea9c913f59e2c5615ea0313e506180d28b2f4e workunit: branch: wip-sjust-testing sha1: a2ea9c913f59e2c5615ea0313e506180d28b2f4e owner: scheduled_sjust@teuthology priority: 90 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - osd.0 - osd.1 - osd.2 - client.0 - client.1 sha1: a2ea9c913f59e2c5615ea0313e506180d28b2f4e suite: rados suite_branch: wip-sjust-testing suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_wip-sjust-testing/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a2ea9c913f59e2c5615ea0313e506180d28b2f4e tasks: - install: null - ceph: conf: global: debug client: 20 debug mds: 20 debug ms: 1 osd max object name len: 460 osd max object namespace len: 64 log-whitelist: - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - exec: client.0: - ceph osd pool create data_cache 4 - ceph osd tier add cephfs_data data_cache - ceph osd tier cache-mode data_cache writeback - ceph osd tier set-overlay cephfs_data data_cache - ceph osd pool set data_cache hit_set_type bloom - ceph osd pool set data_cache hit_set_count 8 - ceph osd pool set data_cache hit_set_period 3600 - ceph osd pool set data_cache min_read_recency_for_promote 0 - ceph-fuse: null - exec: client.0: - sudo chmod 777 $TESTDIR/mnt.0/ - dd if=/dev/urandom of=$TESTDIR/mnt.0/foo bs=1M count=5 - ls -al $TESTDIR/mnt.0/foo - truncate --size 0 $TESTDIR/mnt.0/foo - ls -al $TESTDIR/mnt.0/foo - dd if=/dev/urandom of=$TESTDIR/mnt.0/foo bs=1M count=5 - ls -al $TESTDIR/mnt.0/foo - cp $TESTDIR/mnt.0/foo /tmp/foo - sync - rados -p data_cache ls - - sleep 10 - rados -p data_cache ls - - rados -p data_cache cache-flush-evict-all - rados -p data_cache ls - - sleep 1 - exec: client.1: - hexdump -C /tmp/foo | head - hexdump -C $TESTDIR/mnt.1/foo | head - cmp $TESTDIR/mnt.1/foo /tmp/foo teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.15240 2019-09-28T23:37:07.030 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_wip-sjust-testing/qa; will attempt to use it 2019-09-28T23:37:07.030 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_wip-sjust-testing/qa/tasks 2019-09-28T23:37:07.030 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-09-28T23:37:07.032 INFO:teuthology.task.internal:Checking packages... 2019-09-28T23:37:07.045 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'a2ea9c913f59e2c5615ea0313e506180d28b2f4e' 2019-09-28T23:37:07.045 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-09-28T23:37:07.045 INFO:teuthology.packaging:ref: None 2019-09-28T23:37:07.045 INFO:teuthology.packaging:tag: None 2019-09-28T23:37:07.045 INFO:teuthology.packaging:branch: wip-sjust-testing 2019-09-28T23:37:07.045 INFO:teuthology.packaging:sha1: a2ea9c913f59e2c5615ea0313e506180d28b2f4e 2019-09-28T23:37:07.046 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-sjust-testing 2019-09-28T23:37:07.233 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-5565.ga2ea9c9 2019-09-28T23:37:07.234 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-09-28T23:37:07.245 INFO:teuthology.task.internal:no buildpackages task found 2019-09-28T23:37:07.245 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-09-28T23:37:07.260 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-09-28T23:37:07.260 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-09-28T23:37:07.302 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:37:17.316 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:37:27.330 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:37:37.347 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:37:47.364 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:37:57.381 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:38:07.391 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:38:17.405 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:38:27.461 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:38:37.476 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:38:47.493 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:38:57.510 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:39:07.526 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:39:17.543 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:39:27.556 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:39:37.574 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:39:47.589 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:39:57.604 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:40:07.620 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:40:17.633 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:40:27.650 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:40:37.667 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:40:47.684 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:40:57.701 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:41:07.716 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:41:17.734 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:41:27.752 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:41:37.765 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:41:47.779 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-09-28T23:41:57.796 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_sjust@teuthology', 'description': '/home/teuthworker/archive/sjust-2019-09-28_23:02:01-rados-wip-sjust-testing-distro-basic-smithi/4342933', 'machine_type': 'smithi'} 2019-09-28T23:41:57.808 DEBUG:teuthology.lock.ops:locked ubuntu@smithi084.front.sepia.ceph.com 2019-09-28T23:41:57.869 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-09-28T23:41:58.065 INFO:teuthology.provision.fog.smithi084:Scheduling deploy of centos 7.6 2019-09-28T23:41:58.435 INFO:teuthology.orchestra.console:Power off smithi084 2019-09-28T23:41:58.435 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-09-28T23:41:58.480 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-09-28T23:42:02.520 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-09-28T23:42:06.653 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-09-28T23:42:10.780 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-09-28T23:42:11.009 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2019-09-28T23:42:11.110 INFO:teuthology.orchestra.console:Power on smithi084 2019-09-28T23:42:11.111 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-09-28T23:42:11.152 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-09-28T23:42:15.191 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-09-28T23:42:19.319 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-09-28T23:42:19.545 INFO:teuthology.orchestra.console:Power on for smithi084 completed 2019-09-28T23:42:19.646 INFO:teuthology.provision.fog.smithi084:Waiting for deploy to finish 2019-09-28T23:45:05.650 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi084.front.sepia.ceph.com', 'timeout': 60} 2019-09-28T23:45:14.734 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi084.front.sepia.ceph.com', 'timeout': 60} 2019-09-28T23:45:23.788 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi084.front.sepia.ceph.com', 'timeout': 60} 2019-09-28T23:45:32.849 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi084.front.sepia.ceph.com', 'timeout': 60} 2019-09-28T23:45:41.934 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi084.front.sepia.ceph.com', 'timeout': 60} 2019-09-28T23:45:50.989 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi084.front.sepia.ceph.com', 'timeout': 60} 2019-09-28T23:46:00.045 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi084.front.sepia.ceph.com', 'timeout': 60} 2019-09-28T23:46:09.102 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi084.front.sepia.ceph.com', 'timeout': 60} 2019-09-28T23:46:18.155 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi084.front.sepia.ceph.com', 'timeout': 60} 2019-09-28T23:46:42.605 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi084.front.sepia.ceph.com', 'timeout': 60} 2019-09-28T23:46:51.661 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi084.front.sepia.ceph.com', 'timeout': 60} 2019-09-28T23:46:51.837 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:46:51.837 INFO:teuthology.orchestra.run.smithi084:> hostname 2019-09-28T23:46:52.407 INFO:teuthology.orchestra.run.smithi084.stdout:smithi141 2019-09-28T23:46:52.408 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:46:52.408 INFO:teuthology.orchestra.run.smithi084:> grep smithi141 /etc/hosts 2019-09-28T23:46:52.479 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.141 smithi141.front.sepia.ceph.com smithi141 2019-09-28T23:46:52.479 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:46:52.480 INFO:teuthology.orchestra.run.smithi084:> sudo hostname smithi084 2019-09-28T23:46:52.572 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:46:52.573 INFO:teuthology.orchestra.run.smithi084:> sudo sed -i -e 's/smithi141/smithi084/g' /etc/hosts 2019-09-28T23:46:52.636 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:46:52.636 INFO:teuthology.orchestra.run.smithi084:> sudo sed -i -e 's/172.21.15.141/172.21.15.84/g' /etc/hosts 2019-09-28T23:46:52.692 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:46:52.692 INFO:teuthology.orchestra.run.smithi084:> sudo sed -i -e 's/smithi141/smithi084/g' /etc/hostname 2019-09-28T23:46:52.750 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:46:52.751 INFO:teuthology.orchestra.run.smithi084:> sudo hostname smithi084 2019-09-28T23:46:52.807 INFO:teuthology.provision.fog.smithi084:Deploy complete! 2019-09-28T23:46:52.809 DEBUG:teuthology.parallel:result is None 2019-09-28T23:46:52.948 INFO:teuthology.lock.keys:Checking smithi084.front.sepia.ceph.com 2019-09-28T23:46:52.949 INFO:teuthology.lock.keys:New key found. Updating... 2019-09-28T23:46:52.969 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2019-09-28T23:46:52.986 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi084.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC357mqhnvMS5orFhFM03bvpa95qLpkfPsPCNq3nxLz5y5aP/uVAzFCmbmEXndP4jC+JrlPWP91++UzYt+1gqHbqX693QbTBUjgv8xlR7DPpgl0CoDeuyn/jfYGnp0o/LV3YIOjdRV0WGynPNE/YzTr6FEyr8aYI7JXcjtuYoSu5l2YQf2JYjBn2wIynLgWGDxiSg0dKG0EgZiPIkig2D+a5fPOLV1/CiOkqo522/b9Q1n6V4fSw7Dcwa/SYvSjaUvTJPIMSbzrRm9ng6MBHzx4hyefCH1OPMKTCunOqTx6OxRWPiaNcdXxfNQI98z+iUBolp3mM5v2245V6GhBMApL 2019-09-28T23:46:52.986 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-09-28T23:46:53.036 INFO:teuthology.run_tasks:Running task internal.save_config... 2019-09-28T23:46:53.075 INFO:teuthology.task.internal:Saving configuration 2019-09-28T23:46:53.089 INFO:teuthology.run_tasks:Running task internal.check_lock... 2019-09-28T23:46:53.098 INFO:teuthology.task.internal.check_lock:Checking locks... 2019-09-28T23:46:53.112 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-09-28 23:41:57.802273', u'locked_by': u'scheduled_sjust@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi084.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC357mqhnvMS5orFhFM03bvpa95qLpkfPsPCNq3nxLz5y5aP/uVAzFCmbmEXndP4jC+JrlPWP91++UzYt+1gqHbqX693QbTBUjgv8xlR7DPpgl0CoDeuyn/jfYGnp0o/LV3YIOjdRV0WGynPNE/YzTr6FEyr8aYI7JXcjtuYoSu5l2YQf2JYjBn2wIynLgWGDxiSg0dKG0EgZiPIkig2D+a5fPOLV1/CiOkqo522/b9Q1n6V4fSw7Dcwa/SYvSjaUvTJPIMSbzrRm9ng6MBHzx4hyefCH1OPMKTCunOqTx6OxRWPiaNcdXxfNQI98z+iUBolp3mM5v2245V6GhBMApL', u'description': u'/home/teuthworker/archive/sjust-2019-09-28_23:02:01-rados-wip-sjust-testing-distro-basic-smithi/4342933'} 2019-09-28T23:46:53.112 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2019-09-28T23:46:53.123 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1'] 2019-09-28T23:46:53.124 INFO:teuthology.run_tasks:Running task console_log... 2019-09-28T23:46:53.203 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-09-28T23:46:53.204 INFO:teuthology.run_tasks:Running task internal.connect... 2019-09-28T23:46:53.215 INFO:teuthology.task.internal:Opening connections... 2019-09-28T23:46:53.215 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi084.front.sepia.ceph.com 2019-09-28T23:46:53.217 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi084.front.sepia.ceph.com', 'timeout': 60} 2019-09-28T23:46:53.320 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2019-09-28T23:46:53.331 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:46:53.331 INFO:teuthology.orchestra.run.smithi084:> uname -m 2019-09-28T23:46:53.637 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2019-09-28T23:46:53.638 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:46:53.638 INFO:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2019-09-28T23:46:53.682 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Linux" 2019-09-28T23:46:53.683 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="7 (Core)" 2019-09-28T23:46:53.683 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos" 2019-09-28T23:46:53.683 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora" 2019-09-28T23:46:53.683 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="7" 2019-09-28T23:46:53.683 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-09-28T23:46:53.683 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2019-09-28T23:46:53.684 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-09-28T23:46:53.684 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://www.centos.org/" 2019-09-28T23:46:53.684 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-09-28T23:46:53.684 INFO:teuthology.orchestra.run.smithi084.stdout: 2019-09-28T23:46:53.684 INFO:teuthology.orchestra.run.smithi084.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-09-28T23:46:53.684 INFO:teuthology.orchestra.run.smithi084.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-09-28T23:46:53.685 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-09-28T23:46:53.685 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-09-28T23:46:53.685 INFO:teuthology.orchestra.run.smithi084.stdout: 2019-09-28T23:46:53.686 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2019-09-28T23:46:53.698 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2019-09-28T23:46:53.715 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2019-09-28T23:46:53.743 INFO:teuthology.task.internal:Checking for old test directory... 2019-09-28T23:46:53.743 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:46:53.743 INFO:teuthology.orchestra.run.smithi084:> test '!' -e /home/ubuntu/cephtest 2019-09-28T23:46:53.800 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2019-09-28T23:46:53.810 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2019-09-28T23:46:53.810 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:46:53.810 INFO:teuthology.orchestra.run.smithi084:> test -z $(ls -A /var/lib/ceph) 2019-09-28T23:46:53.862 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2019-09-28T23:46:53.887 INFO:teuthology.run_tasks:Running task kernel... 2019-09-28T23:46:53.910 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mds.a': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.1': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2019-09-28T23:46:53.910 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2019-09-28T23:46:53.912 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@smithi084.front.sepia.ceph.com 2019-09-28T23:46:53.914 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:46:53.915 INFO:teuthology.orchestra.run.smithi084:> sudo package-cleanup -y --oldkernels 2019-09-28T23:46:54.724 INFO:teuthology.orchestra.run.smithi084.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-09-28T23:46:57.832 INFO:teuthology.orchestra.run.smithi084.stdout:No old kernels to remove 2019-09-28T23:46:57.845 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:46:57.845 INFO:teuthology.orchestra.run.smithi084:> uname -r 2019-09-28T23:46:57.890 INFO:teuthology.orchestra.run.smithi084.stdout:3.10.0-957.27.2.el7.x86_64 2019-09-28T23:46:57.890 INFO:teuthology.task.kernel:Running kernel on smithi084: 3.10.0-957.27.2.el7.x86_64 2019-09-28T23:46:57.890 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:46:57.891 INFO:teuthology.orchestra.run.smithi084:> sudo yum install -y kernel 2019-09-28T23:46:58.051 INFO:teuthology.orchestra.run.smithi084.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-09-28T23:46:58.085 INFO:teuthology.orchestra.run.smithi084.stdout:Determining fastest mirrors 2019-09-28T23:46:58.291 INFO:teuthology.orchestra.run.smithi084.stdout: * base: mirror.rackspace.com 2019-09-28T23:46:58.291 INFO:teuthology.orchestra.run.smithi084.stdout: * epel: ftp.linux.ncsu.edu 2019-09-28T23:46:58.292 INFO:teuthology.orchestra.run.smithi084.stdout: * extras: mirror.ash.fastserv.com 2019-09-28T23:46:58.292 INFO:teuthology.orchestra.run.smithi084.stdout: * updates: packages.oit.ncsu.edu 2019-09-28T23:47:00.529 INFO:teuthology.orchestra.run.smithi084.stdout:1 packages excluded due to repository priority protections 2019-09-28T23:47:00.863 INFO:teuthology.orchestra.run.smithi084.stdout:Resolving Dependencies 2019-09-28T23:47:00.863 INFO:teuthology.orchestra.run.smithi084.stdout:--> Running transaction check 2019-09-28T23:47:00.864 INFO:teuthology.orchestra.run.smithi084.stdout:---> Package kernel.x86_64 0:3.10.0-1062.1.1.el7 will be installed 2019-09-28T23:47:01.093 INFO:teuthology.orchestra.run.smithi084.stdout:--> Processing Dependency: linux-firmware >= 20190429-72 for package: kernel-3.10.0-1062.1.1.el7.x86_64 2019-09-28T23:47:01.175 INFO:teuthology.orchestra.run.smithi084.stdout:--> Running transaction check 2019-09-28T23:47:01.175 INFO:teuthology.orchestra.run.smithi084.stdout:---> Package linux-firmware.noarch 0:20180911-69.git85c5d90.el7 will be updated 2019-09-28T23:47:01.203 INFO:teuthology.orchestra.run.smithi084.stdout:---> Package linux-firmware.noarch 0:20190429-72.gitddde598.el7 will be an update 2019-09-28T23:47:02.124 INFO:teuthology.orchestra.run.smithi084.stdout:--> Finished Dependency Resolution 2019-09-28T23:47:02.694 INFO:teuthology.orchestra.run.smithi084.stdout: 2019-09-28T23:47:02.694 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies Resolved 2019-09-28T23:47:02.701 INFO:teuthology.orchestra.run.smithi084.stdout: 2019-09-28T23:47:02.701 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2019-09-28T23:47:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2019-09-28T23:47:02.702 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2019-09-28T23:47:02.702 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2019-09-28T23:47:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: kernel x86_64 3.10.0-1062.1.1.el7 updates 50 M 2019-09-28T23:47:02.702 INFO:teuthology.orchestra.run.smithi084.stdout:Updating for dependencies: 2019-09-28T23:47:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: linux-firmware noarch 20190429-72.gitddde598.el7 base 73 M 2019-09-28T23:47:02.702 INFO:teuthology.orchestra.run.smithi084.stdout: 2019-09-28T23:47:02.703 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2019-09-28T23:47:02.703 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2019-09-28T23:47:02.703 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2019-09-28T23:47:02.703 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrade ( 1 Dependent package) 2019-09-28T23:47:02.703 INFO:teuthology.orchestra.run.smithi084.stdout: 2019-09-28T23:47:02.703 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 123 M 2019-09-28T23:47:02.704 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading packages: 2019-09-28T23:47:02.704 INFO:teuthology.orchestra.run.smithi084.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-09-28T23:47:04.001 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2019-09-28T23:47:04.002 INFO:teuthology.orchestra.run.smithi084.stdout:Total 95 MB/s | 123 MB 00:01 2019-09-28T23:47:04.076 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2019-09-28T23:47:04.274 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2019-09-28T23:47:05.125 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded 2019-09-28T23:47:05.125 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2019-09-28T23:47:17.515 INFO:teuthology.orchestra.run.smithi084.stdout: Updating : linux-firmware-20190429-72.gitddde598.el7.noarch 1/3 2019-09-28T23:47:22.046 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-3.10.0-1062.1.1.el7.x86_64 2/3 2019-09-28T23:47:48.794 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2019-09-28T23:47:48.917 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : linux-firmware-20190429-72.gitddde598.el7.noarch 1/3 2019-09-28T23:47:49.051 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-3.10.0-1062.1.1.el7.x86_64 2/3 2019-09-28T23:47:49.423 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2019-09-28T23:47:49.423 INFO:teuthology.orchestra.run.smithi084.stdout: 2019-09-28T23:47:49.423 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2019-09-28T23:47:49.423 INFO:teuthology.orchestra.run.smithi084.stdout: kernel.x86_64 0:3.10.0-1062.1.1.el7 2019-09-28T23:47:49.424 INFO:teuthology.orchestra.run.smithi084.stdout: 2019-09-28T23:47:49.424 INFO:teuthology.orchestra.run.smithi084.stdout:Dependency Updated: 2019-09-28T23:47:49.424 INFO:teuthology.orchestra.run.smithi084.stdout: linux-firmware.noarch 0:20190429-72.gitddde598.el7 2019-09-28T23:47:49.424 INFO:teuthology.orchestra.run.smithi084.stdout: 2019-09-28T23:47:49.424 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2019-09-28T23:47:49.539 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:47:49.539 INFO:teuthology.orchestra.run.smithi084:> rpm -q kernel --last | head -n 1 2019-09-28T23:47:49.927 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-3.10.0-1062.1.1.el7.x86_64 Sat 28 Sep 2019 11:47:17 PM UTC 2019-09-28T23:47:49.931 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.1.1.el7.x86_64 2019-09-28T23:47:49.931 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-957.27.2.el7.x86_64 Expected: 3.10.0-1062.1.1.el7.x86_64 2019-09-28T23:47:49.931 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2019-09-28T23:47:49.932 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-09-28T23:47:49.932 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2019-09-28T23:47:49.932 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi084.front.sepia.ceph.com, path=None, version=distro) 2019-09-28T23:47:49.933 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:47:49.933 INFO:teuthology.orchestra.run.smithi084:> rpm -q kernel --last | head -n 1 2019-09-28T23:47:50.222 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-3.10.0-1062.1.1.el7.x86_64 Sat 28 Sep 2019 11:47:17 PM UTC 2019-09-28T23:47:50.224 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.1.1.el7.x86_64 2019-09-28T23:47:50.224 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:47:50.224 INFO:teuthology.orchestra.run.smithi084:> sudo rpm -qi grub2 2019-09-28T23:47:50.341 INFO:teuthology.orchestra.run.smithi084.stdout:Name : grub2 2019-09-28T23:47:50.341 INFO:teuthology.orchestra.run.smithi084.stdout:Epoch : 1 2019-09-28T23:47:50.341 INFO:teuthology.orchestra.run.smithi084.stdout:Version : 2.02 2019-09-28T23:47:50.341 INFO:teuthology.orchestra.run.smithi084.stdout:Release : 0.76.el7.centos 2019-09-28T23:47:50.341 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: x86_64 2019-09-28T23:47:50.342 INFO:teuthology.orchestra.run.smithi084.stdout:Install Date: Wed 14 Aug 2019 05:05:05 PM UTC 2019-09-28T23:47:50.342 INFO:teuthology.orchestra.run.smithi084.stdout:Group : System Environment/Base 2019-09-28T23:47:50.342 INFO:teuthology.orchestra.run.smithi084.stdout:Size : 0 2019-09-28T23:47:50.342 INFO:teuthology.orchestra.run.smithi084.stdout:License : GPLv3+ 2019-09-28T23:47:50.342 INFO:teuthology.orchestra.run.smithi084.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:06 PM UTC, Key ID 24c6a8a7f4a80eb5 2019-09-28T23:47:50.342 INFO:teuthology.orchestra.run.smithi084.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2019-09-28T23:47:50.342 INFO:teuthology.orchestra.run.smithi084.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2019-09-28T23:47:50.343 INFO:teuthology.orchestra.run.smithi084.stdout:Build Host : kbuilder.bsys.centos.org 2019-09-28T23:47:50.343 INFO:teuthology.orchestra.run.smithi084.stdout:Relocations : (not relocatable) 2019-09-28T23:47:50.343 INFO:teuthology.orchestra.run.smithi084.stdout:Packager : CentOS BuildSystem 2019-09-28T23:47:50.343 INFO:teuthology.orchestra.run.smithi084.stdout:Vendor : CentOS 2019-09-28T23:47:50.343 INFO:teuthology.orchestra.run.smithi084.stdout:URL : http://www.gnu.org/software/grub/ 2019-09-28T23:47:50.343 INFO:teuthology.orchestra.run.smithi084.stdout:Summary : Bootloader with support for Linux, Multiboot and more 2019-09-28T23:47:50.344 INFO:teuthology.orchestra.run.smithi084.stdout:Description : 2019-09-28T23:47:50.344 INFO:teuthology.orchestra.run.smithi084.stdout: 2019-09-28T23:47:50.344 INFO:teuthology.orchestra.run.smithi084.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2019-09-28T23:47:50.344 INFO:teuthology.orchestra.run.smithi084.stdout:customizable bootloader with modular architecture. It supports a rich 2019-09-28T23:47:50.344 INFO:teuthology.orchestra.run.smithi084.stdout:variety of kernel formats, file systems, computer architectures and 2019-09-28T23:47:50.344 INFO:teuthology.orchestra.run.smithi084.stdout:hardware devices. 2019-09-28T23:47:50.345 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2019-09-28T23:47:50.345 INFO:teuthology.task.kernel:Updating grub on smithi084 to boot 3.10.0-1062.1.1.el7.x86_64 2019-09-28T23:47:50.345 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:47:50.346 INFO:teuthology.orchestra.run.smithi084:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2019-09-28T23:47:50.785 INFO:teuthology.orchestra.run.smithi084.stderr:Generating grub configuration file ... 2019-09-28T23:47:50.953 INFO:teuthology.orchestra.run.smithi084.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.1.1.el7.x86_64 2019-09-28T23:47:50.978 INFO:teuthology.orchestra.run.smithi084.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.1.1.el7.x86_64.img 2019-09-28T23:47:51.225 INFO:teuthology.orchestra.run.smithi084.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.27.2.el7.x86_64 2019-09-28T23:47:51.250 INFO:teuthology.orchestra.run.smithi084.stderr:Found initrd image: /boot/initramfs-3.10.0-957.27.2.el7.x86_64.img 2019-09-28T23:47:51.284 INFO:teuthology.orchestra.run.smithi084.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2019-09-28T23:47:51.301 INFO:teuthology.orchestra.run.smithi084.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2019-09-28T23:47:51.329 INFO:teuthology.orchestra.run.smithi084.stderr:Found linux image: /boot/vmlinuz-0-rescue-48f603b7a56f4bb39f72becf12f3b725 2019-09-28T23:47:51.338 INFO:teuthology.orchestra.run.smithi084.stderr:Found initrd image: /boot/initramfs-0-rescue-48f603b7a56f4bb39f72becf12f3b725.img 2019-09-28T23:47:52.433 INFO:teuthology.orchestra.run.smithi084.stderr:done 2019-09-28T23:47:52.435 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:47:52.435 INFO:teuthology.orchestra.run.smithi084:> mktemp 2019-09-28T23:47:52.514 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.3X2G2Ilvvp 2019-09-28T23:47:52.515 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:47:52.515 INFO:teuthology.orchestra.run.smithi084:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3X2G2Ilvvp 2019-09-28T23:47:52.601 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:47:52.601 INFO:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.3X2G2Ilvvp 2019-09-28T23:47:52.743 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.3X2G2Ilvvp is 6KB 2019-09-28T23:47:52.818 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:47:52.818 INFO:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.3X2G2Ilvvp 2019-09-28T23:47:52.881 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:47:52.881 INFO:teuthology.orchestra.run.smithi084:> sudo grub2-set-default 0 2019-09-28T23:47:53.132 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:47:53.132 INFO:teuthology.orchestra.run.smithi084:> sudo shutdown -r now 2019-09-28T23:47:53.136 INFO:teuthology.misc:Re-opening connections... 2019-09-28T23:47:53.137 INFO:teuthology.misc:trying to connect to ubuntu@smithi084.front.sepia.ceph.com 2019-09-28T23:47:53.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-09-28T23:47:53.139 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi084.front.sepia.ceph.com', 'timeout': 60} 2019-09-28T23:47:53.239 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:47:53.239 INFO:teuthology.orchestra.run.smithi084:> true 2019-09-28T23:47:53.559 DEBUG:teuthology.misc:waited 0.422032117844 2019-09-28T23:47:54.559 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2019-09-28T23:47:54.560 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "3.10.0-1062.1.1.el7.x86_64"... 2019-09-28T23:47:54.560 INFO:teuthology.orchestra.run.smithi084:Running: 2019-09-28T23:47:54.560 INFO:teuthology.orchestra.run.smithi084:> uname -r 2019-09-29T11:37:19.880 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2019-09-29T11:37:19.902 DEBUG:teuthology.task.console_log:Killing console logger for smithi084 2019-09-29T11:37:19.903 DEBUG:teuthology.task.console_log:Killing console logger for smithi084 2019-09-29T11:37:19.903 DEBUG:teuthology.exit:Finished running handlers