2019-11-25T11:17:55.287 INFO:root:teuthology version: 1.0.0-c5aab4e 2019-11-25T11:17:55.287 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.g5cTO4.tmp -- --verbose --lock --description rados:singleton/{all/osd-recovery-incomplete.yaml msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} --name ideepika-2019-11-25_10:38:51-rados:singleton-master-distro-basic-smithi --block --owner scheduled_ideepika@teuthology --archive /home/teuthworker/archive/ideepika-2019-11-25_10:38:51-rados:singleton-master-distro-basic-smithi/4542246 2019-11-25T11:17:55.315 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-11-25T11:17:55.365 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/ideepika-2019-11-25_10:38:51-rados:singleton-master-distro-basic-smithi/4542246 branch: master description: rados:singleton/{all/osd-recovery-incomplete.yaml msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} email: dupadhya@redhat.com first_in_suite: false job_id: '4542246' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: ideepika-2019-11-25_10:38:51-rados:singleton-master-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: '7.6' overrides: admin_socket: branch: master ceph: conf: global: mon client directed command retry: 5 mon client hunt interval max multiple: 2 mon mgr beacon grace: 90 ms bind msgr2: false ms inject socket failures: 1000 ms type: async mgr: debug monc: 10 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME mon_bind_msgr2: false sha1: d7821ce8ec6899cbf9f789c8df4d1921c0052e1f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: d7821ce8ec6899cbf9f789c8df4d1921c0052e1f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-deepika-testing sha1: 56d95fdb7adcd3aab16f52fb2a3ca29d80a5f630 owner: scheduled_ideepika@teuthology priority: 150 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 sha1: d7821ce8ec6899cbf9f789c8df4d1921c0052e1f suite: rados:singleton suite_branch: wip-deepika-testing suite_path: /home/teuthworker/src/github.com_ideepika_ceph_wip-deepika-testing/qa suite_relpath: qa suite_repo: https://github.com/ideepika/ceph.git suite_sha1: 56d95fdb7adcd3aab16f52fb2a3ca29d80a5f630 tasks: - install: null - ceph: conf: osd: osd min pg log entries: 5 osd_fast_fail_on_connection_refused: false log-whitelist: - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - osd_recovery.test_incomplete_pgs: null teuthology_branch: master tube: smithi verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.9281 2019-11-25T11:17:55.365 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ideepika_ceph_wip-deepika-testing/qa; will attempt to use it 2019-11-25T11:17:55.367 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ideepika_ceph_wip-deepika-testing/qa/tasks 2019-11-25T11:17:55.368 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-11-25T11:17:55.369 INFO:teuthology.task.internal:Checking packages... 2019-11-25T11:17:55.382 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'd7821ce8ec6899cbf9f789c8df4d1921c0052e1f' 2019-11-25T11:17:55.382 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-11-25T11:17:55.382 INFO:teuthology.packaging:ref: None 2019-11-25T11:17:55.382 INFO:teuthology.packaging:tag: None 2019-11-25T11:17:55.383 INFO:teuthology.packaging:branch: master 2019-11-25T11:17:55.383 INFO:teuthology.packaging:sha1: d7821ce8ec6899cbf9f789c8df4d1921c0052e1f 2019-11-25T11:17:55.383 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=master 2019-11-25T11:17:56.514 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-7678.gd7821ce 2019-11-25T11:17:56.514 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-11-25T11:17:56.522 INFO:teuthology.task.internal:no buildpackages task found 2019-11-25T11:17:56.522 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-11-25T11:17:56.534 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-11-25T11:17:56.535 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-11-25T11:17:56.575 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-25T11:18:06.587 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-25T11:18:16.610 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-25T11:18:26.628 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-25T11:18:36.641 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-25T11:18:46.654 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-25T11:18:56.670 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-25T11:19:06.679 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-25T11:19:16.697 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-25T11:19:26.713 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-25T11:19:36.729 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-25T11:19:46.747 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-25T11:19:56.765 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-25T11:20:06.782 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-25T11:20:16.797 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_ideepika@teuthology', 'description': '/home/teuthworker/archive/ideepika-2019-11-25_10:38:51-rados:singleton-master-distro-basic-smithi/4542246', 'machine_type': 'smithi'} 2019-11-25T11:20:16.811 DEBUG:teuthology.lock.ops:locked ubuntu@smithi068.front.sepia.ceph.com 2019-11-25T11:20:16.877 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-11-25T11:20:17.057 INFO:teuthology.provision.fog.smithi068:Scheduling deploy of centos 7.6 2019-11-25T11:20:17.352 INFO:teuthology.orchestra.console:Power off smithi068 2019-11-25T11:20:17.352 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi068.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-11-25T11:20:17.393 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi068.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-25T11:20:21.439 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi068.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-25T11:20:25.568 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi068.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-25T11:20:29.700 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi068.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-25T11:20:29.927 INFO:teuthology.orchestra.console:Power off for smithi068 completed 2019-11-25T11:20:30.028 INFO:teuthology.orchestra.console:Power on smithi068 2019-11-25T11:20:30.028 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi068.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-11-25T11:20:30.067 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi068.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-25T11:20:34.107 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi068.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-25T11:20:34.341 INFO:teuthology.orchestra.console:Power on for smithi068 completed 2019-11-25T11:20:34.449 INFO:teuthology.provision.fog.smithi068:Waiting for deploy to finish 2019-11-25T11:23:35.406 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi068.front.sepia.ceph.com', 'timeout': 60} 2019-11-25T11:23:44.460 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi068.front.sepia.ceph.com', 'timeout': 60} 2019-11-25T11:23:53.518 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi068.front.sepia.ceph.com', 'timeout': 60} 2019-11-25T11:24:02.573 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi068.front.sepia.ceph.com', 'timeout': 60} 2019-11-25T11:24:11.630 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi068.front.sepia.ceph.com', 'timeout': 60} 2019-11-25T11:24:20.686 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi068.front.sepia.ceph.com', 'timeout': 60} 2019-11-25T11:24:29.738 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi068.front.sepia.ceph.com', 'timeout': 60} 2019-11-25T11:24:38.795 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi068.front.sepia.ceph.com', 'timeout': 60} 2019-11-25T11:24:47.854 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi068.front.sepia.ceph.com', 'timeout': 60} 2019-11-25T11:24:56.914 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi068.front.sepia.ceph.com', 'timeout': 60} 2019-11-25T11:25:05.998 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi068.front.sepia.ceph.com', 'timeout': 60} 2019-11-25T11:25:43.602 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi068.front.sepia.ceph.com', 'timeout': 60} 2019-11-25T11:25:43.839 INFO:teuthology.orchestra.run.smithi068:> hostname 2019-11-25T11:25:44.310 INFO:teuthology.orchestra.run.smithi068.stdout:smithi068 2019-11-25T11:25:44.311 INFO:teuthology.orchestra.run.smithi068:> grep smithi068 /etc/hosts 2019-11-25T11:25:44.366 INFO:teuthology.orchestra.run.smithi068.stdout:172.21.15.68 smithi068.front.sepia.ceph.com smithi068 2019-11-25T11:25:44.366 INFO:teuthology.orchestra.run.smithi068:> sudo hostname smithi068 2019-11-25T11:25:44.479 INFO:teuthology.orchestra.run.smithi068:> sudo sed -i -e 's/smithi068/smithi068/g' /etc/hosts 2019-11-25T11:25:44.553 INFO:teuthology.orchestra.run.smithi068:> sudo sed -i -e 's/172.21.15.68/172.21.15.68/g' /etc/hosts 2019-11-25T11:25:44.612 INFO:teuthology.orchestra.run.smithi068:> sudo sed -i -e 's/smithi068/smithi068/g' /etc/hostname 2019-11-25T11:25:44.671 INFO:teuthology.orchestra.run.smithi068:> sudo hostname smithi068 2019-11-25T11:25:44.729 INFO:teuthology.provision.fog.smithi068:Deploy complete! 2019-11-25T11:25:44.731 DEBUG:teuthology.parallel:result is None 2019-11-25T11:25:44.895 INFO:teuthology.lock.keys:Checking smithi068.front.sepia.ceph.com 2019-11-25T11:25:44.895 INFO:teuthology.lock.keys:New key found. Updating... 2019-11-25T11:25:44.925 INFO:teuthology.lock.ops:Updating [smithi068.front.sepia.ceph.com]: set os type and version on server 2019-11-25T11:25:44.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-11-25T11:25:44.926 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi068.front.sepia.ceph.com', 'timeout': 60} 2019-11-25T11:25:45.020 INFO:teuthology.orchestra.run.smithi068:> true 2019-11-25T11:25:45.300 INFO:teuthology.orchestra.run.smithi068:> uname -m 2019-11-25T11:25:45.345 INFO:teuthology.orchestra.run.smithi068.stdout:x86_64 2019-11-25T11:25:45.346 INFO:teuthology.orchestra.run.smithi068:> cat /etc/os-release 2019-11-25T11:25:45.389 INFO:teuthology.orchestra.run.smithi068.stdout:NAME="CentOS Linux" 2019-11-25T11:25:45.389 INFO:teuthology.orchestra.run.smithi068.stdout:VERSION="7 (Core)" 2019-11-25T11:25:45.389 INFO:teuthology.orchestra.run.smithi068.stdout:ID="centos" 2019-11-25T11:25:45.390 INFO:teuthology.orchestra.run.smithi068.stdout:ID_LIKE="rhel fedora" 2019-11-25T11:25:45.390 INFO:teuthology.orchestra.run.smithi068.stdout:VERSION_ID="7" 2019-11-25T11:25:45.390 INFO:teuthology.orchestra.run.smithi068.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-11-25T11:25:45.390 INFO:teuthology.orchestra.run.smithi068.stdout:ANSI_COLOR="0;31" 2019-11-25T11:25:45.390 INFO:teuthology.orchestra.run.smithi068.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-11-25T11:25:45.391 INFO:teuthology.orchestra.run.smithi068.stdout:HOME_URL="https://www.centos.org/" 2019-11-25T11:25:45.391 INFO:teuthology.orchestra.run.smithi068.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-11-25T11:25:45.391 INFO:teuthology.orchestra.run.smithi068.stdout: 2019-11-25T11:25:45.391 INFO:teuthology.orchestra.run.smithi068.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-11-25T11:25:45.391 INFO:teuthology.orchestra.run.smithi068.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-11-25T11:25:45.391 INFO:teuthology.orchestra.run.smithi068.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-11-25T11:25:45.391 INFO:teuthology.orchestra.run.smithi068.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-11-25T11:25:45.392 INFO:teuthology.orchestra.run.smithi068.stdout: 2019-11-25T11:25:45.393 INFO:teuthology.lock.ops:Updating smithi068.front.sepia.ceph.com on lock server 2019-11-25T11:25:45.408 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2019-11-25T11:25:45.424 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi068.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGI42z9iB0sovSCpv8BC2026GkVGAviW+AmzifIbvl0P4mvHdmAq95B1XVE/MNPmIxaCb85Q0u9rcKbN97ZoymOz9GBNiu78BWEb3ozWabUkRix87qAZ+ncN0pCgvHnQYRlR38swi6rA5iA5D4ZPL6p8It7URSFjHex4mP4r5B7A5WS/4TpU4KAWYk0/COgt3r3RDRJxHuiMFnAmdSPyvllTJ6nHQ0q4db9SyUMZlIC0Vcj9njo3gVlh4H+gMni74dj3R1oJy3DHrEdg8pG98SsWirzMZ1XiIB6wbDnApMzyFU3S2inCqu2/Mlf4AZ+RelPri06RUMtMbjq0gBRYad 2019-11-25T11:25:45.424 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-11-25T11:25:45.462 INFO:teuthology.run_tasks:Running task internal.save_config... 2019-11-25T11:25:45.471 INFO:teuthology.task.internal:Saving configuration 2019-11-25T11:25:45.487 INFO:teuthology.run_tasks:Running task internal.check_lock... 2019-11-25T11:25:45.494 INFO:teuthology.task.internal.check_lock:Checking locks... 2019-11-25T11:25:45.511 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-11-25 11:20:16.802481', u'locked_by': u'scheduled_ideepika@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi068.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKl3V3xlF19Nodk8MtDxg80+xPL4IOxk1pyaDAsyLuIuG8CBEq3vCUtyol0r9HgB3XSSpSXvhdZJGmMHBFvBphg=', u'description': u'/home/teuthworker/archive/ideepika-2019-11-25_10:38:51-rados:singleton-master-distro-basic-smithi/4542246'} 2019-11-25T11:25:45.511 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2019-11-25T11:25:45.567 INFO:teuthology.task.internal:roles: ubuntu@smithi068.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2019-11-25T11:25:45.568 INFO:teuthology.run_tasks:Running task console_log... 2019-11-25T11:25:45.646 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-11-25T11:25:45.647 INFO:teuthology.run_tasks:Running task internal.connect... 2019-11-25T11:25:45.657 INFO:teuthology.task.internal:Opening connections... 2019-11-25T11:25:45.658 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi068.front.sepia.ceph.com 2019-11-25T11:25:45.659 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi068.front.sepia.ceph.com', 'timeout': 60} 2019-11-25T11:25:45.745 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2019-11-25T11:25:45.801 INFO:teuthology.orchestra.run.smithi068:> uname -m 2019-11-25T11:25:45.965 INFO:teuthology.orchestra.run.smithi068.stdout:x86_64 2019-11-25T11:25:45.965 INFO:teuthology.orchestra.run.smithi068:> cat /etc/os-release 2019-11-25T11:25:46.009 INFO:teuthology.orchestra.run.smithi068.stdout:NAME="CentOS Linux" 2019-11-25T11:25:46.010 INFO:teuthology.orchestra.run.smithi068.stdout:VERSION="7 (Core)" 2019-11-25T11:25:46.010 INFO:teuthology.orchestra.run.smithi068.stdout:ID="centos" 2019-11-25T11:25:46.010 INFO:teuthology.orchestra.run.smithi068.stdout:ID_LIKE="rhel fedora" 2019-11-25T11:25:46.010 INFO:teuthology.orchestra.run.smithi068.stdout:VERSION_ID="7" 2019-11-25T11:25:46.010 INFO:teuthology.orchestra.run.smithi068.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-11-25T11:25:46.011 INFO:teuthology.orchestra.run.smithi068.stdout:ANSI_COLOR="0;31" 2019-11-25T11:25:46.011 INFO:teuthology.orchestra.run.smithi068.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-11-25T11:25:46.011 INFO:teuthology.orchestra.run.smithi068.stdout:HOME_URL="https://www.centos.org/" 2019-11-25T11:25:46.011 INFO:teuthology.orchestra.run.smithi068.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-11-25T11:25:46.011 INFO:teuthology.orchestra.run.smithi068.stdout: 2019-11-25T11:25:46.011 INFO:teuthology.orchestra.run.smithi068.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-11-25T11:25:46.012 INFO:teuthology.orchestra.run.smithi068.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-11-25T11:25:46.012 INFO:teuthology.orchestra.run.smithi068.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-11-25T11:25:46.012 INFO:teuthology.orchestra.run.smithi068.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-11-25T11:25:46.012 INFO:teuthology.orchestra.run.smithi068.stdout: 2019-11-25T11:25:46.012 INFO:teuthology.lock.ops:Updating smithi068.front.sepia.ceph.com on lock server 2019-11-25T11:25:46.035 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2019-11-25T11:25:46.098 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2019-11-25T11:25:46.108 INFO:teuthology.task.internal:Checking for old test directory... 2019-11-25T11:25:46.109 INFO:teuthology.orchestra.run.smithi068:> test '!' -e /home/ubuntu/cephtest 2019-11-25T11:25:46.148 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2019-11-25T11:25:46.158 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2019-11-25T11:25:46.158 INFO:teuthology.orchestra.run.smithi068:> test -z $(ls -A /var/lib/ceph) 2019-11-25T11:25:46.233 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2019-11-25T11:25:46.260 INFO:teuthology.run_tasks:Running task kernel... 2019-11-25T11:25:46.279 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2019-11-25T11:25:46.279 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2019-11-25T11:25:46.279 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@smithi068.front.sepia.ceph.com 2019-11-25T11:25:46.279 INFO:teuthology.orchestra.run.smithi068:> sudo package-cleanup -y --oldkernels 2019-11-25T11:25:47.117 INFO:teuthology.orchestra.run.smithi068.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-11-25T11:25:48.129 INFO:teuthology.orchestra.run.smithi068.stdout:http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/733703470dd10accd9234492acb0c74de09f91772c2767107f6b0fa567cbc26a-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2019-11-25T11:25:48.129 INFO:teuthology.orchestra.run.smithi068.stdout:Trying other mirror. 2019-11-25T11:25:48.129 INFO:teuthology.orchestra.run.smithi068.stdout:To address this issue please refer to the below wiki article 2019-11-25T11:25:48.129 INFO:teuthology.orchestra.run.smithi068.stdout: 2019-11-25T11:25:48.129 INFO:teuthology.orchestra.run.smithi068.stdout:https://wiki.centos.org/yum-errors 2019-11-25T11:25:48.129 INFO:teuthology.orchestra.run.smithi068.stdout: 2019-11-25T11:25:48.130 INFO:teuthology.orchestra.run.smithi068.stdout:If above article doesn't help to resolve this issue please use https://bugs.centos.org/. 2019-11-25T11:25:48.130 INFO:teuthology.orchestra.run.smithi068.stdout: 2019-11-25T11:25:48.275 INFO:teuthology.orchestra.run.smithi068.stdout:http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/733703470dd10accd9234492acb0c74de09f91772c2767107f6b0fa567cbc26a-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2019-11-25T11:25:48.276 INFO:teuthology.orchestra.run.smithi068.stdout:Trying other mirror. 2019-11-25T11:25:48.366 INFO:teuthology.orchestra.run.smithi068.stdout:http://mirror.oss.ou.edu/epel/7/x86_64/repodata/733703470dd10accd9234492acb0c74de09f91772c2767107f6b0fa567cbc26a-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2019-11-25T11:25:48.366 INFO:teuthology.orchestra.run.smithi068.stdout:Trying other mirror. 2019-11-25T11:25:48.510 INFO:teuthology.orchestra.run.smithi068.stdout:http://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/733703470dd10accd9234492acb0c74de09f91772c2767107f6b0fa567cbc26a-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2019-11-25T11:25:48.510 INFO:teuthology.orchestra.run.smithi068.stdout:Trying other mirror. 2019-11-25T11:25:48.662 INFO:teuthology.orchestra.run.smithi068.stdout:http://mirror.pnl.gov/epel/7/x86_64/repodata/733703470dd10accd9234492acb0c74de09f91772c2767107f6b0fa567cbc26a-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2019-11-25T11:25:48.663 INFO:teuthology.orchestra.run.smithi068.stdout:Trying other mirror. 2019-11-25T11:25:49.250 INFO:teuthology.orchestra.run.smithi068.stdout:No old kernels to remove 2019-11-25T11:25:49.266 INFO:teuthology.orchestra.run.smithi068:> uname -r 2019-11-25T11:25:49.319 INFO:teuthology.orchestra.run.smithi068.stdout:3.10.0-1062.4.1.el7.x86_64 2019-11-25T11:25:49.320 INFO:teuthology.task.kernel:Running kernel on smithi068: 3.10.0-1062.4.1.el7.x86_64 2019-11-25T11:25:49.320 INFO:teuthology.orchestra.run.smithi068:> sudo yum install -y kernel 2019-11-25T11:25:49.502 INFO:teuthology.orchestra.run.smithi068.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-11-25T11:25:49.539 INFO:teuthology.orchestra.run.smithi068.stdout:Determining fastest mirrors 2019-11-25T11:25:49.689 INFO:teuthology.orchestra.run.smithi068.stdout: * base: mirror.es.its.nyu.edu 2019-11-25T11:25:49.690 INFO:teuthology.orchestra.run.smithi068.stdout: * epel: ftp.linux.ncsu.edu 2019-11-25T11:25:49.690 INFO:teuthology.orchestra.run.smithi068.stdout: * extras: mirror.es.its.nyu.edu 2019-11-25T11:25:49.690 INFO:teuthology.orchestra.run.smithi068.stdout: * updates: packages.oit.ncsu.edu 2019-11-25T11:25:50.957 INFO:teuthology.orchestra.run.smithi068.stdout:Resolving Dependencies 2019-11-25T11:25:50.957 INFO:teuthology.orchestra.run.smithi068.stdout:--> Running transaction check 2019-11-25T11:25:50.957 INFO:teuthology.orchestra.run.smithi068.stdout:---> Package kernel.x86_64 0:3.10.0-1062.4.3.el7 will be installed 2019-11-25T11:25:52.146 INFO:teuthology.orchestra.run.smithi068.stdout:--> Finished Dependency Resolution 2019-11-25T11:25:52.864 INFO:teuthology.orchestra.run.smithi068.stdout: 2019-11-25T11:25:52.864 INFO:teuthology.orchestra.run.smithi068.stdout:Dependencies Resolved 2019-11-25T11:25:52.865 INFO:teuthology.orchestra.run.smithi068.stdout: 2019-11-25T11:25:52.866 INFO:teuthology.orchestra.run.smithi068.stdout:================================================================================ 2019-11-25T11:25:52.866 INFO:teuthology.orchestra.run.smithi068.stdout: Package Arch Version Repository Size 2019-11-25T11:25:52.866 INFO:teuthology.orchestra.run.smithi068.stdout:================================================================================ 2019-11-25T11:25:52.866 INFO:teuthology.orchestra.run.smithi068.stdout:Installing: 2019-11-25T11:25:52.867 INFO:teuthology.orchestra.run.smithi068.stdout: kernel x86_64 3.10.0-1062.4.3.el7 updates 50 M 2019-11-25T11:25:52.867 INFO:teuthology.orchestra.run.smithi068.stdout: 2019-11-25T11:25:52.867 INFO:teuthology.orchestra.run.smithi068.stdout:Transaction Summary 2019-11-25T11:25:52.868 INFO:teuthology.orchestra.run.smithi068.stdout:================================================================================ 2019-11-25T11:25:52.868 INFO:teuthology.orchestra.run.smithi068.stdout:Install 1 Package 2019-11-25T11:25:52.869 INFO:teuthology.orchestra.run.smithi068.stdout: 2019-11-25T11:25:52.869 INFO:teuthology.orchestra.run.smithi068.stdout:Total download size: 50 M 2019-11-25T11:25:52.869 INFO:teuthology.orchestra.run.smithi068.stdout:Installed size: 64 M 2019-11-25T11:25:52.870 INFO:teuthology.orchestra.run.smithi068.stdout:Downloading packages: 2019-11-25T11:25:52.873 INFO:teuthology.orchestra.run.smithi068.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-11-25T11:25:53.761 INFO:teuthology.orchestra.run.smithi068.stdout:Running transaction check 2019-11-25T11:25:53.951 INFO:teuthology.orchestra.run.smithi068.stdout:Running transaction test 2019-11-25T11:25:54.816 INFO:teuthology.orchestra.run.smithi068.stdout:Transaction test succeeded 2019-11-25T11:25:54.816 INFO:teuthology.orchestra.run.smithi068.stdout:Running transaction 2019-11-25T11:26:31.759 INFO:teuthology.orchestra.run.smithi068.stdout: Installing : kernel-3.10.0-1062.4.3.el7.x86_64 1/1 2019-11-25T11:26:32.571 INFO:teuthology.orchestra.run.smithi068.stdout: Verifying : kernel-3.10.0-1062.4.3.el7.x86_64 1/1 2019-11-25T11:26:32.571 INFO:teuthology.orchestra.run.smithi068.stdout: 2019-11-25T11:26:32.571 INFO:teuthology.orchestra.run.smithi068.stdout:Installed: 2019-11-25T11:26:32.571 INFO:teuthology.orchestra.run.smithi068.stdout: kernel.x86_64 0:3.10.0-1062.4.3.el7 2019-11-25T11:26:32.572 INFO:teuthology.orchestra.run.smithi068.stdout: 2019-11-25T11:26:32.572 INFO:teuthology.orchestra.run.smithi068.stdout:Complete! 2019-11-25T11:26:32.621 INFO:teuthology.orchestra.run.smithi068:> rpm -q kernel --last | head -n 1 2019-11-25T11:26:33.012 INFO:teuthology.orchestra.run.smithi068.stdout:kernel-3.10.0-1062.4.3.el7.x86_64 Mon 25 Nov 2019 11:25:58 AM UTC 2019-11-25T11:26:33.015 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.4.3.el7.x86_64 2019-11-25T11:26:33.016 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-1062.4.1.el7.x86_64 Expected: 3.10.0-1062.4.3.el7.x86_64 2019-11-25T11:26:33.016 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2019-11-25T11:26:33.016 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-11-25T11:26:33.016 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2019-11-25T11:26:33.016 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi068.front.sepia.ceph.com, path=None, version=distro) 2019-11-25T11:26:33.016 INFO:teuthology.orchestra.run.smithi068:> rpm -q kernel --last | head -n 1 2019-11-25T11:26:33.272 INFO:teuthology.orchestra.run.smithi068.stdout:kernel-3.10.0-1062.4.3.el7.x86_64 Mon 25 Nov 2019 11:25:58 AM UTC 2019-11-25T11:26:33.273 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.4.3.el7.x86_64 2019-11-25T11:26:33.273 INFO:teuthology.orchestra.run.smithi068:> sudo rpm -qi grub2 2019-11-25T11:26:33.445 INFO:teuthology.orchestra.run.smithi068.stdout:Name : grub2 2019-11-25T11:26:33.446 INFO:teuthology.orchestra.run.smithi068.stdout:Epoch : 1 2019-11-25T11:26:33.446 INFO:teuthology.orchestra.run.smithi068.stdout:Version : 2.02 2019-11-25T11:26:33.446 INFO:teuthology.orchestra.run.smithi068.stdout:Release : 0.76.el7.centos 2019-11-25T11:26:33.446 INFO:teuthology.orchestra.run.smithi068.stdout:Architecture: x86_64 2019-11-25T11:26:33.446 INFO:teuthology.orchestra.run.smithi068.stdout:Install Date: Tue 05 Nov 2019 03:41:14 PM UTC 2019-11-25T11:26:33.446 INFO:teuthology.orchestra.run.smithi068.stdout:Group : System Environment/Base 2019-11-25T11:26:33.447 INFO:teuthology.orchestra.run.smithi068.stdout:Size : 0 2019-11-25T11:26:33.447 INFO:teuthology.orchestra.run.smithi068.stdout:License : GPLv3+ 2019-11-25T11:26:33.447 INFO:teuthology.orchestra.run.smithi068.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:06 PM UTC, Key ID 24c6a8a7f4a80eb5 2019-11-25T11:26:33.447 INFO:teuthology.orchestra.run.smithi068.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2019-11-25T11:26:33.447 INFO:teuthology.orchestra.run.smithi068.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2019-11-25T11:26:33.447 INFO:teuthology.orchestra.run.smithi068.stdout:Build Host : kbuilder.bsys.centos.org 2019-11-25T11:26:33.447 INFO:teuthology.orchestra.run.smithi068.stdout:Relocations : (not relocatable) 2019-11-25T11:26:33.447 INFO:teuthology.orchestra.run.smithi068.stdout:Packager : CentOS BuildSystem 2019-11-25T11:26:33.448 INFO:teuthology.orchestra.run.smithi068.stdout:Vendor : CentOS 2019-11-25T11:26:33.448 INFO:teuthology.orchestra.run.smithi068.stdout:URL : http://www.gnu.org/software/grub/ 2019-11-25T11:26:33.448 INFO:teuthology.orchestra.run.smithi068.stdout:Summary : Bootloader with support for Linux, Multiboot and more 2019-11-25T11:26:33.448 INFO:teuthology.orchestra.run.smithi068.stdout:Description : 2019-11-25T11:26:33.448 INFO:teuthology.orchestra.run.smithi068.stdout: 2019-11-25T11:26:33.448 INFO:teuthology.orchestra.run.smithi068.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2019-11-25T11:26:33.448 INFO:teuthology.orchestra.run.smithi068.stdout:customizable bootloader with modular architecture. It supports a rich 2019-11-25T11:26:33.449 INFO:teuthology.orchestra.run.smithi068.stdout:variety of kernel formats, file systems, computer architectures and 2019-11-25T11:26:33.449 INFO:teuthology.orchestra.run.smithi068.stdout:hardware devices. 2019-11-25T11:26:33.450 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2019-11-25T11:26:33.450 INFO:teuthology.task.kernel:Updating grub on smithi068 to boot 3.10.0-1062.4.3.el7.x86_64 2019-11-25T11:26:33.450 INFO:teuthology.orchestra.run.smithi068:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2019-11-25T11:26:33.968 INFO:teuthology.orchestra.run.smithi068.stderr:Generating grub configuration file ... 2019-11-25T11:26:34.123 INFO:teuthology.orchestra.run.smithi068.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.4.3.el7.x86_64 2019-11-25T11:26:34.140 INFO:teuthology.orchestra.run.smithi068.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.4.3.el7.x86_64.img 2019-11-25T11:26:34.453 INFO:teuthology.orchestra.run.smithi068.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.4.1.el7.x86_64 2019-11-25T11:26:34.485 INFO:teuthology.orchestra.run.smithi068.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.4.1.el7.x86_64.img 2019-11-25T11:26:34.528 INFO:teuthology.orchestra.run.smithi068.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2019-11-25T11:26:34.550 INFO:teuthology.orchestra.run.smithi068.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2019-11-25T11:26:34.588 INFO:teuthology.orchestra.run.smithi068.stderr:Found linux image: /boot/vmlinuz-0-rescue-a1739c9c60b14f0891126d2e603bb003 2019-11-25T11:26:34.600 INFO:teuthology.orchestra.run.smithi068.stderr:Found initrd image: /boot/initramfs-0-rescue-a1739c9c60b14f0891126d2e603bb003.img 2019-11-25T11:26:35.587 INFO:teuthology.orchestra.run.smithi068.stderr:done 2019-11-25T11:26:35.590 INFO:teuthology.orchestra.run.smithi068:> mktemp 2019-11-25T11:26:35.660 INFO:teuthology.orchestra.run.smithi068.stdout:/tmp/tmp.gynJmSn97T 2019-11-25T11:26:35.661 INFO:teuthology.orchestra.run.smithi068:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gynJmSn97T 2019-11-25T11:26:35.751 INFO:teuthology.orchestra.run.smithi068:> sudo chmod 0666 /tmp/tmp.gynJmSn97T 2019-11-25T11:26:35.941 DEBUG:teuthology.orchestra.remote:smithi068:/tmp/tmp.gynJmSn97T is 6KB 2019-11-25T11:26:36.015 INFO:teuthology.orchestra.run.smithi068:> rm -fr /tmp/tmp.gynJmSn97T 2019-11-25T11:26:36.092 INFO:teuthology.orchestra.run.smithi068:> sudo grub2-set-default 0 2019-11-25T11:26:36.387 INFO:teuthology.orchestra.run.smithi068:> sudo shutdown -r now 2019-11-25T11:26:36.392 INFO:teuthology.misc:Re-opening connections... 2019-11-25T11:26:36.393 INFO:teuthology.misc:trying to connect to ubuntu@smithi068.front.sepia.ceph.com 2019-11-25T11:26:36.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-11-25T11:26:36.395 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi068.front.sepia.ceph.com', 'timeout': 60} 2019-11-25T11:26:36.496 INFO:teuthology.orchestra.run.smithi068:> true 2019-11-25T11:26:36.849 DEBUG:teuthology.misc:waited 0.456257104874 2019-11-25T11:26:37.850 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2019-11-25T11:26:37.850 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "3.10.0-1062.4.3.el7.x86_64"... 2019-11-25T11:26:37.851 INFO:teuthology.orchestra.run.smithi068:> uname -r 2019-11-25T23:19:28.341 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2019-11-25T23:19:28.360 DEBUG:teuthology.task.console_log:Killing console logger for smithi068 2019-11-25T23:19:28.360 DEBUG:teuthology.task.console_log:Killing console logger for smithi068 2019-11-25T23:19:28.361 DEBUG:teuthology.exit:Finished running handlers