2019-11-06T02:25:48.344 INFO:root:teuthology version: 1.0.0-99c249e 2019-11-06T02:25:48.345 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.dUUYGc.tmp -- --verbose --lock --description rados/singleton-nomsgr/{all/health-warnings.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} --name kchai-2019-11-05_12:08:55-rados-wip-kefu-testing-2019-11-03-1250-distro-basic-mira --block --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2019-11-05_12:08:55-rados-wip-kefu-testing-2019-11-03-1250-distro-basic-mira/4474456 2019-11-06T02:25:48.369 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-11-06T02:25:48.423 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2019-11-05_12:08:55-rados-wip-kefu-testing-2019-11-03-1250-distro-basic-mira/4474456 branch: wip-kefu-testing-2019-11-03-1250 description: rados/singleton-nomsgr/{all/health-warnings.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} email: tchaikov@gmail.com first_in_suite: false job_id: '4474456' kernel: kdb: true sha1: distro last_in_suite: false machine_type: mira name: kchai-2019-11-05_12:08:55-rados-wip-kefu-testing-2019-11-03-1250-distro-basic-mira nuke-on-error: true os_type: centos os_version: '7.6' overrides: admin_socket: branch: wip-kefu-testing-2019-11-03-1250 ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 osd debug verify cached snaps: true osd debug verify missing on start: true osd op queue: debug_random osd op queue cut off: debug_random log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 4bad7f756c2b0609ac17bae7e818ce6a17d3b451 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 4bad7f756c2b0609ac17bae7e818ce6a17d3b451 workunit: branch: wip-kefu-testing-2019-11-03-1250 sha1: 4bad7f756c2b0609ac17bae7e818ce6a17d3b451 owner: scheduled_kchai@teuthology priority: 99 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 - osd.4 - osd.5 - osd.6 - osd.7 - osd.8 - osd.9 - client.0 sha1: 4bad7f756c2b0609ac17bae7e818ce6a17d3b451 suite: rados suite_branch: wip-kefu-testing-2019-11-03-1250 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-11-03-1250/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 4bad7f756c2b0609ac17bae7e818ce6a17d3b451 tasks: - install: null - ceph: conf: osd: osd max object name len: 400 osd max object namespace len: 64 log-whitelist: - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - workunit: clients: all: - rados/test_health_warnings.sh teuthology_branch: master tube: mira verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.mira.11883 2019-11-06T02:25:48.423 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-11-03-1250/qa; will attempt to use it 2019-11-06T02:25:48.424 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-11-03-1250/qa/tasks 2019-11-06T02:25:48.424 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-11-06T02:25:48.426 INFO:teuthology.task.internal:Checking packages... 2019-11-06T02:25:48.438 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '4bad7f756c2b0609ac17bae7e818ce6a17d3b451' 2019-11-06T02:25:48.439 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-11-06T02:25:48.439 INFO:teuthology.packaging:ref: None 2019-11-06T02:25:48.439 INFO:teuthology.packaging:tag: None 2019-11-06T02:25:48.439 INFO:teuthology.packaging:branch: wip-kefu-testing-2019-11-03-1250 2019-11-06T02:25:48.439 INFO:teuthology.packaging:sha1: 4bad7f756c2b0609ac17bae7e818ce6a17d3b451 2019-11-06T02:25:48.439 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-kefu-testing-2019-11-03-1250 2019-11-06T02:25:48.603 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-6775.g4bad7f7 2019-11-06T02:25:48.604 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-11-06T02:25:48.614 INFO:teuthology.task.internal:no buildpackages task found 2019-11-06T02:25:48.614 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-11-06T02:25:48.677 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-11-06T02:25:48.678 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-11-06T02:25:48.720 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_kchai@teuthology', 'description': '/home/teuthworker/archive/kchai-2019-11-05_12:08:55-rados-wip-kefu-testing-2019-11-03-1250-distro-basic-mira/4474456', 'machine_type': 'mira'} 2019-11-06T02:25:48.733 DEBUG:teuthology.lock.ops:locked ubuntu@mira109.front.sepia.ceph.com 2019-11-06T02:25:48.791 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-11-06T02:25:48.982 INFO:teuthology.provision.fog.mira109:Scheduling deploy of centos 7.6 2019-11-06T02:25:49.306 INFO:teuthology.orchestra.console:Power off mira109 2019-11-06T02:25:49.306 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira109.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-11-06T02:25:49.347 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira109.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-06T02:25:53.388 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira109.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-06T02:25:57.520 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira109.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-06T02:25:57.748 INFO:teuthology.orchestra.console:Power off for mira109 completed 2019-11-06T02:25:57.850 INFO:teuthology.orchestra.console:Power on mira109 2019-11-06T02:25:57.850 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira109.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-11-06T02:25:57.894 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira109.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-06T02:26:01.936 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira109.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-06T02:26:06.065 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira109.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-06T02:26:06.298 INFO:teuthology.orchestra.console:Power on for mira109 completed 2019-11-06T02:26:06.399 INFO:teuthology.provision.fog.mira109:Waiting for deploy to finish 2019-11-06T02:28:52.417 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira109.front.sepia.ceph.com', 'timeout': 60} 2019-11-06T02:29:01.484 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira109.front.sepia.ceph.com', 'timeout': 60} 2019-11-06T02:29:10.546 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira109.front.sepia.ceph.com', 'timeout': 60} 2019-11-06T02:29:19.629 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira109.front.sepia.ceph.com', 'timeout': 60} 2019-11-06T02:29:28.685 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira109.front.sepia.ceph.com', 'timeout': 60} 2019-11-06T02:29:37.741 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira109.front.sepia.ceph.com', 'timeout': 60} 2019-11-06T02:30:43.744 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira109.front.sepia.ceph.com', 'timeout': 60} 2019-11-06T02:30:44.417 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:30:44.417 INFO:teuthology.orchestra.run.mira109:> hostname 2019-11-06T02:30:46.132 INFO:teuthology.orchestra.run.mira109.stdout:mira109.front.sepia.ceph.com 2019-11-06T02:30:46.133 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:30:46.133 INFO:teuthology.orchestra.run.mira109:> grep mira109.front.sepia.ceph.com /etc/hosts 2019-11-06T02:30:46.260 INFO:teuthology.orchestra.run.mira109.stdout:172.21.9.140 mira109.front.sepia.ceph.com mira109 2019-11-06T02:30:46.261 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:30:46.261 INFO:teuthology.orchestra.run.mira109:> sudo hostname mira109 2019-11-06T02:30:46.803 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:30:46.803 INFO:teuthology.orchestra.run.mira109:> sudo sed -i -e 's/mira109.front.sepia.ceph.com/mira109/g' /etc/hosts 2019-11-06T02:30:46.933 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:30:46.933 INFO:teuthology.orchestra.run.mira109:> sudo sed -i -e 's/172.21.9.140/172.21.9.140/g' /etc/hosts 2019-11-06T02:30:47.066 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:30:47.066 INFO:teuthology.orchestra.run.mira109:> sudo sed -i -e 's/mira109.front.sepia.ceph.com/mira109/g' /etc/hostname 2019-11-06T02:30:47.227 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:30:47.227 INFO:teuthology.orchestra.run.mira109:> sudo hostname mira109 2019-11-06T02:30:47.392 INFO:teuthology.provision.fog.mira109:Deploy complete! 2019-11-06T02:30:47.395 DEBUG:teuthology.parallel:result is None 2019-11-06T02:30:47.553 INFO:teuthology.lock.keys:Checking mira109.front.sepia.ceph.com 2019-11-06T02:30:47.553 INFO:teuthology.lock.keys:New key found. Updating... 2019-11-06T02:30:47.572 INFO:teuthology.lock.ops:Updating [mira109.front.sepia.ceph.com]: set os type and version on server 2019-11-06T02:30:47.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-11-06T02:30:47.573 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira109.front.sepia.ceph.com', 'timeout': 60} 2019-11-06T02:30:47.678 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:30:47.678 INFO:teuthology.orchestra.run.mira109:> true 2019-11-06T02:30:48.148 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:30:48.148 INFO:teuthology.orchestra.run.mira109:> uname -m 2019-11-06T02:30:48.265 INFO:teuthology.orchestra.run.mira109.stdout:x86_64 2019-11-06T02:30:48.266 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:30:48.266 INFO:teuthology.orchestra.run.mira109:> cat /etc/os-release 2019-11-06T02:30:48.378 INFO:teuthology.orchestra.run.mira109.stdout:NAME="CentOS Linux" 2019-11-06T02:30:48.378 INFO:teuthology.orchestra.run.mira109.stdout:VERSION="7 (Core)" 2019-11-06T02:30:48.379 INFO:teuthology.orchestra.run.mira109.stdout:ID="centos" 2019-11-06T02:30:48.379 INFO:teuthology.orchestra.run.mira109.stdout:ID_LIKE="rhel fedora" 2019-11-06T02:30:48.379 INFO:teuthology.orchestra.run.mira109.stdout:VERSION_ID="7" 2019-11-06T02:30:48.379 INFO:teuthology.orchestra.run.mira109.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-11-06T02:30:48.379 INFO:teuthology.orchestra.run.mira109.stdout:ANSI_COLOR="0;31" 2019-11-06T02:30:48.379 INFO:teuthology.orchestra.run.mira109.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-11-06T02:30:48.379 INFO:teuthology.orchestra.run.mira109.stdout:HOME_URL="https://www.centos.org/" 2019-11-06T02:30:48.380 INFO:teuthology.orchestra.run.mira109.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-11-06T02:30:48.380 INFO:teuthology.orchestra.run.mira109.stdout: 2019-11-06T02:30:48.380 INFO:teuthology.orchestra.run.mira109.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-11-06T02:30:48.380 INFO:teuthology.orchestra.run.mira109.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-11-06T02:30:48.380 INFO:teuthology.orchestra.run.mira109.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-11-06T02:30:48.380 INFO:teuthology.orchestra.run.mira109.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-11-06T02:30:48.380 INFO:teuthology.orchestra.run.mira109.stdout: 2019-11-06T02:30:48.381 INFO:teuthology.lock.ops:Updating mira109.front.sepia.ceph.com on lock server 2019-11-06T02:30:48.399 INFO:teuthology.task.internal.lock_machines:1 mira machines locked this try, 1/1 locked so far 2019-11-06T02:30:48.416 INFO:teuthology.task.internal.lock_machines:Locked targets: mira109.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDW5dHT3q3BPAck4XkcxdPxacI0bt4rl0fdEJsEfXSUG0yVzOm4jKY0PJpeU0x5yOv9oLuGzfLer/rEoSeKlux2dL+kM4l9aMX7FAmXN8yW7rxt1ZdbHToMdO5uNPcvPwHbyKzbVYxIPwxJSG4OKm2qicEDSgUTzbLP6AUzEr1KA74fR1TPiNb5r0jdxyCVmMXxyExGXN4eow4HmUb0oYcjabRqRxB9ekUIJobuWC9YM9nj/YxUZSR936DehL7zAo/pymQpN1NcaUxjvYZRmeEw+HPr7DNnTnnDHWINkkvjY+H40gEtcmeo2wYcQSQ9UfR16bjAX26Xn4JHE6L8Kxvn 2019-11-06T02:30:48.416 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-11-06T02:30:48.460 INFO:teuthology.run_tasks:Running task internal.save_config... 2019-11-06T02:30:48.471 INFO:teuthology.task.internal:Saving configuration 2019-11-06T02:30:48.483 INFO:teuthology.run_tasks:Running task internal.check_lock... 2019-11-06T02:30:48.493 INFO:teuthology.task.internal.check_lock:Checking locks... 2019-11-06T02:30:48.509 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-11-06 02:25:48.727128', u'locked_by': u'scheduled_kchai@teuthology', u'up': True, u'mac_address': u'00:25:90:0d:1f:f8', u'name': u'mira109.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'mira', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJN/NsfU8d31K1GgrS05g3vVQ08HaG59ENMX9F7u3OqiVo0f+fQaxF9A6MvNJH61mK/53KdtKNNoTUtDmXtm7R8=', u'description': u'/home/teuthworker/archive/kchai-2019-11-05_12:08:55-rados-wip-kefu-testing-2019-11-03-1250-distro-basic-mira/4474456'} 2019-11-06T02:30:48.509 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2019-11-06T02:30:48.550 INFO:teuthology.task.internal:roles: ubuntu@mira109.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'osd.8', 'osd.9', 'client.0'] 2019-11-06T02:30:48.550 INFO:teuthology.run_tasks:Running task console_log... 2019-11-06T02:30:48.634 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-11-06T02:30:48.634 INFO:teuthology.run_tasks:Running task internal.connect... 2019-11-06T02:30:48.645 INFO:teuthology.task.internal:Opening connections... 2019-11-06T02:30:48.645 DEBUG:teuthology.task.internal:connecting to ubuntu@mira109.front.sepia.ceph.com 2019-11-06T02:30:48.646 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira109.front.sepia.ceph.com', 'timeout': 60} 2019-11-06T02:30:48.792 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2019-11-06T02:30:48.814 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:30:48.814 INFO:teuthology.orchestra.run.mira109:> uname -m 2019-11-06T02:30:49.197 INFO:teuthology.orchestra.run.mira109.stdout:x86_64 2019-11-06T02:30:49.197 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:30:49.198 INFO:teuthology.orchestra.run.mira109:> cat /etc/os-release 2019-11-06T02:30:49.299 INFO:teuthology.orchestra.run.mira109.stdout:NAME="CentOS Linux" 2019-11-06T02:30:49.299 INFO:teuthology.orchestra.run.mira109.stdout:VERSION="7 (Core)" 2019-11-06T02:30:49.299 INFO:teuthology.orchestra.run.mira109.stdout:ID="centos" 2019-11-06T02:30:49.299 INFO:teuthology.orchestra.run.mira109.stdout:ID_LIKE="rhel fedora" 2019-11-06T02:30:49.299 INFO:teuthology.orchestra.run.mira109.stdout:VERSION_ID="7" 2019-11-06T02:30:49.299 INFO:teuthology.orchestra.run.mira109.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-11-06T02:30:49.300 INFO:teuthology.orchestra.run.mira109.stdout:ANSI_COLOR="0;31" 2019-11-06T02:30:49.300 INFO:teuthology.orchestra.run.mira109.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-11-06T02:30:49.300 INFO:teuthology.orchestra.run.mira109.stdout:HOME_URL="https://www.centos.org/" 2019-11-06T02:30:49.300 INFO:teuthology.orchestra.run.mira109.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-11-06T02:30:49.300 INFO:teuthology.orchestra.run.mira109.stdout: 2019-11-06T02:30:49.300 INFO:teuthology.orchestra.run.mira109.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-11-06T02:30:49.300 INFO:teuthology.orchestra.run.mira109.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-11-06T02:30:49.301 INFO:teuthology.orchestra.run.mira109.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-11-06T02:30:49.301 INFO:teuthology.orchestra.run.mira109.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-11-06T02:30:49.301 INFO:teuthology.orchestra.run.mira109.stdout: 2019-11-06T02:30:49.301 INFO:teuthology.lock.ops:Updating mira109.front.sepia.ceph.com on lock server 2019-11-06T02:30:49.318 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2019-11-06T02:30:49.336 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2019-11-06T02:30:49.346 INFO:teuthology.task.internal:Checking for old test directory... 2019-11-06T02:30:49.346 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:30:49.346 INFO:teuthology.orchestra.run.mira109:> test '!' -e /home/ubuntu/cephtest 2019-11-06T02:30:49.452 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2019-11-06T02:30:49.460 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2019-11-06T02:30:49.460 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:30:49.460 INFO:teuthology.orchestra.run.mira109:> test -z $(ls -A /var/lib/ceph) 2019-11-06T02:30:49.572 INFO:teuthology.orchestra.run.mira109.stderr:ls: cannot access /var/lib/ceph: No such file or directory 2019-11-06T02:30:49.574 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2019-11-06T02:30:49.598 INFO:teuthology.run_tasks:Running task kernel... 2019-11-06T02:30:49.624 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.8': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.5': {'sha1': 'distro', 'kdb': True}, 'osd.4': {'sha1': 'distro', 'kdb': True}, 'osd.7': {'sha1': 'distro', 'kdb': True}, 'osd.6': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}, 'osd.9': {'sha1': 'distro', 'kdb': True}} 2019-11-06T02:30:49.625 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2019-11-06T02:30:49.625 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@mira109.front.sepia.ceph.com 2019-11-06T02:30:49.625 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:30:49.625 INFO:teuthology.orchestra.run.mira109:> sudo package-cleanup -y --oldkernels 2019-11-06T02:30:55.737 INFO:teuthology.orchestra.run.mira109.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-11-06T02:31:06.508 INFO:teuthology.orchestra.run.mira109.stdout:No old kernels to remove 2019-11-06T02:31:06.537 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:31:06.538 INFO:teuthology.orchestra.run.mira109:> uname -r 2019-11-06T02:31:06.665 INFO:teuthology.orchestra.run.mira109.stdout:3.10.0-957.21.3.el7.x86_64 2019-11-06T02:31:06.665 INFO:teuthology.task.kernel:Running kernel on mira109: 3.10.0-957.21.3.el7.x86_64 2019-11-06T02:31:06.666 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:31:06.666 INFO:teuthology.orchestra.run.mira109:> sudo yum install -y kernel 2019-11-06T02:31:07.115 INFO:teuthology.orchestra.run.mira109.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-11-06T02:31:07.193 INFO:teuthology.orchestra.run.mira109.stdout:Determining fastest mirrors 2019-11-06T02:31:10.297 INFO:teuthology.orchestra.run.mira109.stdout: * base: mirror.wdc1.us.leaseweb.net 2019-11-06T02:31:10.298 INFO:teuthology.orchestra.run.mira109.stdout: * epel: ftp.linux.ncsu.edu 2019-11-06T02:31:10.299 INFO:teuthology.orchestra.run.mira109.stdout: * extras: mirror.wdc1.us.leaseweb.net 2019-11-06T02:31:10.300 INFO:teuthology.orchestra.run.mira109.stdout: * updates: mirror.jaleco.com 2019-11-06T02:31:15.583 INFO:teuthology.orchestra.run.mira109.stdout:1 packages excluded due to repository priority protections 2019-11-06T02:31:16.258 INFO:teuthology.orchestra.run.mira109.stdout:Resolving Dependencies 2019-11-06T02:31:16.259 INFO:teuthology.orchestra.run.mira109.stdout:--> Running transaction check 2019-11-06T02:31:16.259 INFO:teuthology.orchestra.run.mira109.stdout:---> Package kernel.x86_64 0:3.10.0-1062.4.1.el7 will be installed 2019-11-06T02:31:17.563 INFO:teuthology.orchestra.run.mira109.stdout:--> Processing Dependency: linux-firmware >= 20190429-72 for package: kernel-3.10.0-1062.4.1.el7.x86_64 2019-11-06T02:31:17.870 INFO:teuthology.orchestra.run.mira109.stdout:--> Running transaction check 2019-11-06T02:31:17.870 INFO:teuthology.orchestra.run.mira109.stdout:---> Package linux-firmware.noarch 0:20180911-69.git85c5d90.el7 will be updated 2019-11-06T02:31:17.883 INFO:teuthology.orchestra.run.mira109.stdout:---> Package linux-firmware.noarch 0:20190429-72.gitddde598.el7 will be an update 2019-11-06T02:31:25.171 INFO:teuthology.orchestra.run.mira109.stdout:--> Finished Dependency Resolution 2019-11-06T02:31:33.472 INFO:teuthology.orchestra.run.mira109.stdout: 2019-11-06T02:31:33.472 INFO:teuthology.orchestra.run.mira109.stdout:Dependencies Resolved 2019-11-06T02:31:33.535 INFO:teuthology.orchestra.run.mira109.stdout: 2019-11-06T02:31:33.535 INFO:teuthology.orchestra.run.mira109.stdout:================================================================================ 2019-11-06T02:31:33.536 INFO:teuthology.orchestra.run.mira109.stdout: Package Arch Version Repository Size 2019-11-06T02:31:33.536 INFO:teuthology.orchestra.run.mira109.stdout:================================================================================ 2019-11-06T02:31:33.536 INFO:teuthology.orchestra.run.mira109.stdout:Installing: 2019-11-06T02:31:33.536 INFO:teuthology.orchestra.run.mira109.stdout: kernel x86_64 3.10.0-1062.4.1.el7 updates 50 M 2019-11-06T02:31:33.536 INFO:teuthology.orchestra.run.mira109.stdout:Updating for dependencies: 2019-11-06T02:31:33.536 INFO:teuthology.orchestra.run.mira109.stdout: linux-firmware noarch 20190429-72.gitddde598.el7 base 73 M 2019-11-06T02:31:33.536 INFO:teuthology.orchestra.run.mira109.stdout: 2019-11-06T02:31:33.536 INFO:teuthology.orchestra.run.mira109.stdout:Transaction Summary 2019-11-06T02:31:33.537 INFO:teuthology.orchestra.run.mira109.stdout:================================================================================ 2019-11-06T02:31:33.537 INFO:teuthology.orchestra.run.mira109.stdout:Install 1 Package 2019-11-06T02:31:33.537 INFO:teuthology.orchestra.run.mira109.stdout:Upgrade ( 1 Dependent package) 2019-11-06T02:31:33.537 INFO:teuthology.orchestra.run.mira109.stdout: 2019-11-06T02:31:33.537 INFO:teuthology.orchestra.run.mira109.stdout:Total download size: 123 M 2019-11-06T02:31:33.537 INFO:teuthology.orchestra.run.mira109.stdout:Downloading packages: 2019-11-06T02:31:33.538 INFO:teuthology.orchestra.run.mira109.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-11-06T02:33:15.175 INFO:teuthology.orchestra.run.mira109.stdout:-------------------------------------------------------------------------------- 2019-11-06T02:33:15.175 INFO:teuthology.orchestra.run.mira109.stdout:Total 1.2 MB/s | 123 MB 01:41 2019-11-06T02:33:15.288 INFO:teuthology.orchestra.run.mira109.stdout:Running transaction check 2019-11-06T02:33:15.620 INFO:teuthology.orchestra.run.mira109.stdout:Running transaction test 2019-11-06T02:33:23.912 INFO:teuthology.orchestra.run.mira109.stdout:Transaction test succeeded 2019-11-06T02:33:23.912 INFO:teuthology.orchestra.run.mira109.stdout:Running transaction 2019-11-06T02:34:04.868 INFO:teuthology.orchestra.run.mira109.stdout: Updating : linux-firmware-20190429-72.gitddde598.el7.noarch 1/3 2019-11-06T02:34:17.257 INFO:teuthology.orchestra.run.mira109.stdout: Installing : kernel-3.10.0-1062.4.1.el7.x86_64 2/3 2019-11-06T02:35:05.286 INFO:teuthology.orchestra.run.mira109.stdout: Cleanup : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2019-11-06T02:35:05.379 INFO:teuthology.orchestra.run.mira109.stdout: Verifying : kernel-3.10.0-1062.4.1.el7.x86_64 1/3 2019-11-06T02:35:05.403 INFO:teuthology.orchestra.run.mira109.stdout: Verifying : linux-firmware-20190429-72.gitddde598.el7.noarch 2/3 2019-11-06T02:35:05.625 INFO:teuthology.orchestra.run.mira109.stdout: Verifying : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2019-11-06T02:35:05.625 INFO:teuthology.orchestra.run.mira109.stdout: 2019-11-06T02:35:05.625 INFO:teuthology.orchestra.run.mira109.stdout:Installed: 2019-11-06T02:35:05.626 INFO:teuthology.orchestra.run.mira109.stdout: kernel.x86_64 0:3.10.0-1062.4.1.el7 2019-11-06T02:35:05.626 INFO:teuthology.orchestra.run.mira109.stdout: 2019-11-06T02:35:05.626 INFO:teuthology.orchestra.run.mira109.stdout:Dependency Updated: 2019-11-06T02:35:05.626 INFO:teuthology.orchestra.run.mira109.stdout: linux-firmware.noarch 0:20190429-72.gitddde598.el7 2019-11-06T02:35:05.626 INFO:teuthology.orchestra.run.mira109.stdout: 2019-11-06T02:35:05.626 INFO:teuthology.orchestra.run.mira109.stdout:Complete! 2019-11-06T02:35:05.738 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:35:05.738 INFO:teuthology.orchestra.run.mira109:> rpm -q kernel --last | head -n 1 2019-11-06T02:35:06.270 INFO:teuthology.orchestra.run.mira109.stdout:kernel-3.10.0-1062.4.1.el7.x86_64 Wed 06 Nov 2019 02:34:04 AM UTC 2019-11-06T02:35:06.273 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.4.1.el7.x86_64 2019-11-06T02:35:06.273 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-957.21.3.el7.x86_64 Expected: 3.10.0-1062.4.1.el7.x86_64 2019-11-06T02:35:06.273 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2019-11-06T02:35:06.273 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-11-06T02:35:06.273 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2019-11-06T02:35:06.274 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@mira109.front.sepia.ceph.com, path=None, version=distro) 2019-11-06T02:35:06.274 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:35:06.274 INFO:teuthology.orchestra.run.mira109:> rpm -q kernel --last | head -n 1 2019-11-06T02:35:06.760 INFO:teuthology.orchestra.run.mira109.stdout:kernel-3.10.0-1062.4.1.el7.x86_64 Wed 06 Nov 2019 02:34:04 AM UTC 2019-11-06T02:35:06.763 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.4.1.el7.x86_64 2019-11-06T02:35:06.763 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:35:06.763 INFO:teuthology.orchestra.run.mira109:> sudo rpm -qi grub2 2019-11-06T02:35:06.981 INFO:teuthology.orchestra.run.mira109.stdout:Name : grub2 2019-11-06T02:35:06.981 INFO:teuthology.orchestra.run.mira109.stdout:Epoch : 1 2019-11-06T02:35:06.981 INFO:teuthology.orchestra.run.mira109.stdout:Version : 2.02 2019-11-06T02:35:06.981 INFO:teuthology.orchestra.run.mira109.stdout:Release : 0.76.el7.centos 2019-11-06T02:35:06.981 INFO:teuthology.orchestra.run.mira109.stdout:Architecture: x86_64 2019-11-06T02:35:06.982 INFO:teuthology.orchestra.run.mira109.stdout:Install Date: Fri 12 Jul 2019 03:03:58 PM UTC 2019-11-06T02:35:06.982 INFO:teuthology.orchestra.run.mira109.stdout:Group : System Environment/Base 2019-11-06T02:35:06.982 INFO:teuthology.orchestra.run.mira109.stdout:Size : 0 2019-11-06T02:35:06.982 INFO:teuthology.orchestra.run.mira109.stdout:License : GPLv3+ 2019-11-06T02:35:06.982 INFO:teuthology.orchestra.run.mira109.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:06 PM UTC, Key ID 24c6a8a7f4a80eb5 2019-11-06T02:35:06.982 INFO:teuthology.orchestra.run.mira109.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2019-11-06T02:35:06.982 INFO:teuthology.orchestra.run.mira109.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2019-11-06T02:35:06.982 INFO:teuthology.orchestra.run.mira109.stdout:Build Host : kbuilder.bsys.centos.org 2019-11-06T02:35:06.983 INFO:teuthology.orchestra.run.mira109.stdout:Relocations : (not relocatable) 2019-11-06T02:35:06.983 INFO:teuthology.orchestra.run.mira109.stdout:Packager : CentOS BuildSystem 2019-11-06T02:35:06.983 INFO:teuthology.orchestra.run.mira109.stdout:Vendor : CentOS 2019-11-06T02:35:06.983 INFO:teuthology.orchestra.run.mira109.stdout:URL : http://www.gnu.org/software/grub/ 2019-11-06T02:35:06.983 INFO:teuthology.orchestra.run.mira109.stdout:Summary : Bootloader with support for Linux, Multiboot and more 2019-11-06T02:35:06.983 INFO:teuthology.orchestra.run.mira109.stdout:Description : 2019-11-06T02:35:06.983 INFO:teuthology.orchestra.run.mira109.stdout: 2019-11-06T02:35:06.983 INFO:teuthology.orchestra.run.mira109.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2019-11-06T02:35:06.984 INFO:teuthology.orchestra.run.mira109.stdout:customizable bootloader with modular architecture. It supports a rich 2019-11-06T02:35:06.984 INFO:teuthology.orchestra.run.mira109.stdout:variety of kernel formats, file systems, computer architectures and 2019-11-06T02:35:06.984 INFO:teuthology.orchestra.run.mira109.stdout:hardware devices. 2019-11-06T02:35:06.985 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2019-11-06T02:35:06.985 INFO:teuthology.task.kernel:Updating grub on mira109 to boot 3.10.0-1062.4.1.el7.x86_64 2019-11-06T02:35:06.985 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:35:06.985 INFO:teuthology.orchestra.run.mira109:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2019-11-06T02:35:07.628 INFO:teuthology.orchestra.run.mira109.stderr:Generating grub configuration file ... 2019-11-06T02:35:07.824 INFO:teuthology.orchestra.run.mira109.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.4.1.el7.x86_64 2019-11-06T02:35:07.852 INFO:teuthology.orchestra.run.mira109.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.4.1.el7.x86_64.img 2019-11-06T02:35:08.371 INFO:teuthology.orchestra.run.mira109.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.21.3.el7.x86_64 2019-11-06T02:35:08.442 INFO:teuthology.orchestra.run.mira109.stderr:Found initrd image: /boot/initramfs-3.10.0-957.21.3.el7.x86_64.img 2019-11-06T02:35:08.513 INFO:teuthology.orchestra.run.mira109.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2019-11-06T02:35:08.546 INFO:teuthology.orchestra.run.mira109.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2019-11-06T02:35:08.609 INFO:teuthology.orchestra.run.mira109.stderr:Found linux image: /boot/vmlinuz-0-rescue-ed19c4d1a216479b9bcb22a6fb70d4b0 2019-11-06T02:35:08.632 INFO:teuthology.orchestra.run.mira109.stderr:Found initrd image: /boot/initramfs-0-rescue-ed19c4d1a216479b9bcb22a6fb70d4b0.img 2019-11-06T02:35:09.812 INFO:teuthology.orchestra.run.mira109.stderr:done 2019-11-06T02:35:09.815 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:35:09.815 INFO:teuthology.orchestra.run.mira109:> mktemp 2019-11-06T02:35:09.933 INFO:teuthology.orchestra.run.mira109.stdout:/tmp/tmp.yH0Vu8P4TY 2019-11-06T02:35:09.933 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:35:09.934 INFO:teuthology.orchestra.run.mira109:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yH0Vu8P4TY 2019-11-06T02:35:10.089 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:35:10.089 INFO:teuthology.orchestra.run.mira109:> sudo chmod 0666 /tmp/tmp.yH0Vu8P4TY 2019-11-06T02:35:10.344 DEBUG:teuthology.orchestra.remote:mira109:/tmp/tmp.yH0Vu8P4TY is 6KB 2019-11-06T02:35:10.449 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:35:10.449 INFO:teuthology.orchestra.run.mira109:> rm -fr /tmp/tmp.yH0Vu8P4TY 2019-11-06T02:35:10.540 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:35:10.540 INFO:teuthology.orchestra.run.mira109:> sudo grub2-set-default 0 2019-11-06T02:35:10.720 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:35:10.720 INFO:teuthology.orchestra.run.mira109:> sudo shutdown -r now 2019-11-06T02:35:10.726 INFO:teuthology.misc:Re-opening connections... 2019-11-06T02:35:10.726 INFO:teuthology.misc:trying to connect to ubuntu@mira109.front.sepia.ceph.com 2019-11-06T02:35:10.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-11-06T02:35:10.728 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira109.front.sepia.ceph.com', 'timeout': 60} 2019-11-06T02:35:10.864 INFO:teuthology.orchestra.run.mira109:Running: 2019-11-06T02:35:10.864 INFO:teuthology.orchestra.run.mira109:> true 2019-11-06T14:26:01.931 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2019-11-06T14:26:01.942 DEBUG:teuthology.task.console_log:Killing console logger for mira109 2019-11-06T14:26:01.942 DEBUG:teuthology.task.console_log:Killing console logger for mira109 2019-11-06T14:26:01.943 DEBUG:teuthology.exit:Finished running handlers