2020-03-27T13:38:52.448 INFO:root:teuthology version: 1.0.0-b3a6b1a 2020-03-27T13:38:52.449 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.4OpThM.tmp -- --verbose --lock --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_latest.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/set-chunks-read.yaml} --name kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi --block --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi/4894903 2020-03-27T13:38:52.737 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-27T13:38:52.822 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi/4894903 branch: nautilus-kefu2-testing-2020-03-27-1709 description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_latest.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/set-chunks-read.yaml} email: tchaikov@gmail.com first_in_suite: false job_id: '4894903' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: centos os_version: '7.5' overrides: admin_socket: branch: nautilus-kefu2-testing-2020-03-27-1709 ceph: conf: global: ms bind msgr2: false ms inject socket failures: 2500 ms tcp read timeout: 5 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 2 osd_recovery_max_active: 10 osd_recovery_max_single_start: 10 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 filestore odsync write: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd backoff on degraded: true osd backoff on peering: true osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 2 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 0.5 fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request mon_bind_msgr2: false sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: nautilus-kefu2-testing-2020-03-27-1709 sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a owner: scheduled_kchai@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a suite: rados suite_branch: nautilus-kefu2-testing-2020-03-27-1709 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-1709/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a tasks: - install: null - ceph: null - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph osd set-require-min-compat-client luminous - ceph balancer mode upmap - ceph balancer on - thrashosds: chance_pgnum_grow: 2 chance_pgpnum_fix: 1 timeout: 1200 - exec: client.0: - sudo ceph osd pool create low_tier 4 - rados: clients: - client.0 low_tier_pool: low_tier objects: 300 op_weights: chunk_read: 100 tier_promote: 10 ops: 4000 set_chunk: true teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.2416 2020-03-27T13:38:52.822 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-1709/qa; will attempt to use it 2020-03-27T13:38:52.823 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-1709/qa/tasks 2020-03-27T13:38:52.823 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-27T13:38:53.095 INFO:teuthology.task.internal:Checking packages... 2020-03-27T13:38:53.122 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'e9aaac6a13136ef54c1c1edf22f45ec35b28b19a' 2020-03-27T13:38:53.123 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-27T13:38:53.123 INFO:teuthology.packaging:ref: None 2020-03-27T13:38:53.123 INFO:teuthology.packaging:tag: None 2020-03-27T13:38:53.123 INFO:teuthology.packaging:branch: nautilus-kefu2-testing-2020-03-27-1709 2020-03-27T13:38:53.123 INFO:teuthology.packaging:sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a 2020-03-27T13:38:53.124 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=nautilus-kefu2-testing-2020-03-27-1709 2020-03-27T13:38:53.349 INFO:teuthology.task.internal:Found packages for ceph version 14.2.8-299.ge9aaac6 2020-03-27T13:38:53.349 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-27T13:38:53.463 INFO:teuthology.task.internal:no buildpackages task found 2020-03-27T13:38:53.463 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-27T13:38:53.539 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-27T13:38:53.539 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-27T13:38:53.599 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-03-27T13:39:03.619 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-03-27T13:39:13.644 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-03-27T13:39:23.663 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-03-27T13:39:33.673 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-03-27T13:39:43.689 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-03-27T13:39:53.709 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-03-27T13:40:03.729 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-03-27T13:40:13.746 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-03-27T13:40:23.763 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-03-27T13:40:33.781 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-03-27T13:40:43.797 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-03-27T13:40:53.813 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-03-27T13:41:03.829 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-27T13:41:13.845 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:41:23.854 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:41:33.875 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:41:43.889 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:41:53.901 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:42:03.910 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:42:13.925 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:42:23.942 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:42:33.957 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:42:43.976 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:42:53.993 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:43:04.011 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:43:14.029 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:43:24.043 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:43:34.061 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:43:44.089 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:43:54.109 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:44:04.121 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:44:14.137 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:44:24.154 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:44:34.171 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:44:44.189 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-27T13:44:54.208 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:45:04.228 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:45:14.247 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:45:24.268 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:45:34.304 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:45:44.325 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:45:54.348 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:46:04.365 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:46:14.383 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:46:24.402 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:46:34.423 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:46:44.442 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-27T13:46:54.463 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:47:04.485 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_kchai@teuthology', 'description': '/home/teuthworker/archive/kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi/4894903', 'machine_type': 'smithi'} 2020-03-27T13:47:04.501 DEBUG:teuthology.lock.ops:locked ubuntu@smithi143.front.sepia.ceph.com, ubuntu@smithi088.front.sepia.ceph.com 2020-03-27T13:47:04.554 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-27T13:47:04.749 INFO:teuthology.provision.fog.smithi088:Scheduling deploy of centos 7.5 2020-03-27T13:47:04.752 INFO:teuthology.provision.fog.smithi143:Scheduling deploy of centos 7.5 2020-03-27T13:47:05.075 INFO:teuthology.orchestra.console:Power off smithi088 2020-03-27T13:47:05.075 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-27T13:47:05.093 INFO:teuthology.orchestra.console:Power off smithi143 2020-03-27T13:47:05.093 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-27T13:47:05.101 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:47:05.113 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:47:09.125 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:47:09.135 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:47:13.237 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:47:13.245 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:47:17.349 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:47:17.359 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:47:17.560 INFO:teuthology.orchestra.console:Power off for smithi088 completed 2020-03-27T13:47:17.568 INFO:teuthology.orchestra.console:Power off for smithi143 completed 2020-03-27T13:47:17.662 INFO:teuthology.orchestra.console:Power on smithi088 2020-03-27T13:47:17.663 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-27T13:47:17.673 INFO:teuthology.orchestra.console:Power on smithi143 2020-03-27T13:47:17.673 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-27T13:47:17.683 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:47:17.692 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:47:21.705 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:47:21.714 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:47:25.817 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:47:25.825 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:47:26.027 INFO:teuthology.orchestra.console:Power on for smithi088 completed 2020-03-27T13:47:26.034 INFO:teuthology.orchestra.console:Power on for smithi143 completed 2020-03-27T13:47:26.129 INFO:teuthology.provision.fog.smithi088:Waiting for deploy to finish 2020-03-27T13:47:26.136 INFO:teuthology.provision.fog.smithi143:Waiting for deploy to finish 2020-03-27T13:49:57.103 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi143.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:50:06.189 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi143.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:50:12.181 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi088.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:50:15.274 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi143.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:50:21.255 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi088.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:50:24.325 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi143.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:50:30.310 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi088.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:50:33.382 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi143.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:50:39.378 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi088.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:50:42.438 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi143.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:50:48.454 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi088.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:50:51.494 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi143.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:50:57.518 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi088.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:51:06.601 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi088.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:51:15.661 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi088.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:51:23.366 INFO:teuthology.orchestra.run.smithi143:> hostname 2020-03-27T13:51:23.817 INFO:teuthology.orchestra.run.smithi143.stdout:smithi124 2020-03-27T13:51:23.818 INFO:teuthology.orchestra.run.smithi143:> grep smithi124 /etc/hosts 2020-03-27T13:51:23.865 INFO:teuthology.orchestra.run.smithi143.stdout:172.21.15.124 smithi124.front.sepia.ceph.com smithi124 2020-03-27T13:51:23.866 INFO:teuthology.orchestra.run.smithi143:> sudo hostname smithi143 2020-03-27T13:51:23.952 INFO:teuthology.orchestra.run.smithi143:> sudo sed -i -e 's/smithi124/smithi143/g' /etc/hosts 2020-03-27T13:51:24.002 INFO:teuthology.orchestra.run.smithi143:> sudo sed -i -e 's/172.21.15.124/172.21.15.143/g' /etc/hosts 2020-03-27T13:51:24.051 INFO:teuthology.orchestra.run.smithi143:> sudo sed -i -e 's/smithi124/smithi143/g' /etc/hostname 2020-03-27T13:51:24.101 INFO:teuthology.orchestra.run.smithi143:> sudo hostname smithi143 2020-03-27T13:51:24.146 INFO:teuthology.provision.fog.smithi143:Deploy complete! 2020-03-27T13:51:24.149 DEBUG:teuthology.parallel:result is None 2020-03-27T13:51:24.742 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi088.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:30.745 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi088.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:30.935 INFO:teuthology.orchestra.run.smithi088:> hostname 2020-03-27T13:52:31.509 INFO:teuthology.orchestra.run.smithi088.stdout:smithi088 2020-03-27T13:52:31.509 INFO:teuthology.orchestra.run.smithi088:> grep smithi088 /etc/hosts 2020-03-27T13:52:31.590 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88 smithi088.front.sepia.ceph.com smithi088 2020-03-27T13:52:31.590 INFO:teuthology.orchestra.run.smithi088:> sudo hostname smithi088 2020-03-27T13:52:31.702 INFO:teuthology.orchestra.run.smithi088:> sudo sed -i -e 's/smithi088/smithi088/g' /etc/hosts 2020-03-27T13:52:31.802 INFO:teuthology.orchestra.run.smithi088:> sudo sed -i -e 's/172.21.15.88/172.21.15.88/g' /etc/hosts 2020-03-27T13:52:31.876 INFO:teuthology.orchestra.run.smithi088:> sudo sed -i -e 's/smithi088/smithi088/g' /etc/hostname 2020-03-27T13:52:31.955 INFO:teuthology.orchestra.run.smithi088:> sudo hostname smithi088 2020-03-27T13:52:32.023 INFO:teuthology.provision.fog.smithi088:Deploy complete! 2020-03-27T13:52:32.024 DEBUG:teuthology.parallel:result is None 2020-03-27T13:52:32.229 INFO:teuthology.lock.ops:Checking smithi088.front.sepia.ceph.com 2020-03-27T13:52:32.229 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-27T13:52:32.245 INFO:teuthology.lock.ops:Checking smithi143.front.sepia.ceph.com 2020-03-27T13:52:32.245 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-27T13:52:32.259 INFO:teuthology.lock.ops:Updating [smithi088.front.sepia.ceph.com]: set os type and version on server 2020-03-27T13:52:32.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-27T13:52:32.260 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi088.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:32.406 INFO:teuthology.orchestra.run.smithi088:> true 2020-03-27T13:52:32.789 INFO:teuthology.orchestra.run.smithi088:> uname -m 2020-03-27T13:52:32.856 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2020-03-27T13:52:32.857 INFO:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2020-03-27T13:52:32.914 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Linux" 2020-03-27T13:52:32.914 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="7 (Core)" 2020-03-27T13:52:32.914 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos" 2020-03-27T13:52:32.915 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora" 2020-03-27T13:52:32.915 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="7" 2020-03-27T13:52:32.916 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-03-27T13:52:32.916 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2020-03-27T13:52:32.916 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-03-27T13:52:32.918 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://www.centos.org/" 2020-03-27T13:52:32.918 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-27T13:52:32.918 INFO:teuthology.orchestra.run.smithi088.stdout: 2020-03-27T13:52:32.919 INFO:teuthology.orchestra.run.smithi088.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-03-27T13:52:32.919 INFO:teuthology.orchestra.run.smithi088.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-03-27T13:52:32.919 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-27T13:52:32.919 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-03-27T13:52:32.919 INFO:teuthology.orchestra.run.smithi088.stdout: 2020-03-27T13:52:32.920 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2020-03-27T13:52:32.939 INFO:teuthology.lock.ops:Updating [smithi143.front.sepia.ceph.com]: set os type and version on server 2020-03-27T13:52:32.939 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-27T13:52:32.939 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi143.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:33.052 INFO:teuthology.orchestra.run.smithi143:> true 2020-03-27T13:52:33.444 INFO:teuthology.orchestra.run.smithi143:> uname -m 2020-03-27T13:52:33.529 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2020-03-27T13:52:33.529 INFO:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2020-03-27T13:52:33.608 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Linux" 2020-03-27T13:52:33.608 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="7 (Core)" 2020-03-27T13:52:33.608 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos" 2020-03-27T13:52:33.608 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora" 2020-03-27T13:52:33.608 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="7" 2020-03-27T13:52:33.608 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-03-27T13:52:33.609 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31" 2020-03-27T13:52:33.609 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-03-27T13:52:33.609 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://www.centos.org/" 2020-03-27T13:52:33.609 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-27T13:52:33.609 INFO:teuthology.orchestra.run.smithi143.stdout: 2020-03-27T13:52:33.609 INFO:teuthology.orchestra.run.smithi143.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-03-27T13:52:33.609 INFO:teuthology.orchestra.run.smithi143.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-03-27T13:52:33.610 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-27T13:52:33.610 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-03-27T13:52:33.610 INFO:teuthology.orchestra.run.smithi143.stdout: 2020-03-27T13:52:33.610 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2020-03-27T13:52:33.633 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-03-27T13:52:33.678 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi088.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1oSI7VP2Oe2EBDIABcpvNtQ+/0mEtueIWgJOekYbjfB770iGwSGCvcrr9uX1uOWb0ZtjqA+DSr5o6YirAmGR5LPmUIc8jHbEdyYNjTVrLi92tjEAcd3Ezy9Sv9BAR0ZjKunA+k49VD8QSs/PyXxqXL/JiliQt7XXruZnUZ0yXyznfcNC6twQnDHdqHwJ3VUG5TJKomqu797p0+rdBcYwa/bMK5LZK4ewgZ9coXPbYt3hRyWWrI2cEp6h1ZpVkSMHv3v5B/kJ8WSs8YeX8ZjJh3DRKSuSbOvu3U3GTen8T4/v5GfgMNnreHsR2CKUqzIEp2ed8djEumyfT2yKnQF6/ smithi143.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC3tIZFX7ymxZqJ3KzJez2k7R42qcSX022Wkt7egmEe9Higk0U6QSwfjLXxcaE2oGP+UCIlN3wGgH/SrhYQb4cR9scGPr2XTcdE6yYSS/ImIS4CI9tmjzbeMrluVwi1KSGzeBoIx6pHTMnDVhF6171ink+J8UsNYu/iUQdAJKIlMNjTLkGqMtNapuwYNG4odEQ62KbUqP5Pz7Jp8WqFwnbGe7y+HP8P2MlHq77LGmp2bP7lB15NghlmGrgogNUY48TpNWciFIytBai2Q91yGHzGU6c24dCnaexxRetpA3KLmTyACj3u9GTc4pJwZUUVtDSkd3sdPLVctEFb76b/D/fT 2020-03-27T13:52:33.678 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-27T13:52:33.729 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-03-27T13:52:33.999 INFO:teuthology.task.internal:Saving configuration 2020-03-27T13:52:34.020 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-03-27T13:52:34.144 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-03-27T13:52:34.162 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-27 13:47:04.487415', u'locked_by': u'scheduled_kchai@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi088.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLMx5wwH8jRHf5LmMVxJrIzWPqKBa9iEvbI9ShS7uokf1ZHRF7/3KeBvgIVEENGJU6VKb99zfdS9VaQ8v/u7QC4=', u'description': u'/home/teuthworker/archive/kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi/4894903'} 2020-03-27T13:52:34.183 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-27 13:47:04.487436', u'locked_by': u'scheduled_kchai@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi143.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNB7ZP6sOq6E308Gj9Ss25mHsSUKMxQ54TFh4T7JgZbI/ISzhuGrGfB505gWNnFkF4ZhuZT+S8SAwo+mgBsftKQ=', u'description': u'/home/teuthworker/archive/kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi/4894903'} 2020-03-27T13:52:34.183 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-03-27T13:52:34.207 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2020-03-27T13:52:34.207 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2020-03-27T13:52:34.207 INFO:teuthology.run_tasks:Running task console_log... 2020-03-27T13:52:34.345 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-27T13:52:34.346 INFO:teuthology.run_tasks:Running task internal.connect... 2020-03-27T13:52:34.401 INFO:teuthology.task.internal:Opening connections... 2020-03-27T13:52:34.401 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2020-03-27T13:52:34.402 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi088.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:34.509 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2020-03-27T13:52:34.510 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi143.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:34.617 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-03-27T13:52:34.624 INFO:teuthology.orchestra.run.smithi088:> uname -m 2020-03-27T13:52:34.845 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2020-03-27T13:52:34.846 INFO:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2020-03-27T13:52:34.907 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Linux" 2020-03-27T13:52:34.908 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="7 (Core)" 2020-03-27T13:52:34.908 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos" 2020-03-27T13:52:34.908 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora" 2020-03-27T13:52:34.908 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="7" 2020-03-27T13:52:34.908 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-03-27T13:52:34.908 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2020-03-27T13:52:34.909 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-03-27T13:52:34.909 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://www.centos.org/" 2020-03-27T13:52:34.909 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-27T13:52:34.909 INFO:teuthology.orchestra.run.smithi088.stdout: 2020-03-27T13:52:34.909 INFO:teuthology.orchestra.run.smithi088.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-03-27T13:52:34.909 INFO:teuthology.orchestra.run.smithi088.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-03-27T13:52:34.909 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-27T13:52:34.909 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-03-27T13:52:34.910 INFO:teuthology.orchestra.run.smithi088.stdout: 2020-03-27T13:52:34.910 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2020-03-27T13:52:34.928 INFO:teuthology.orchestra.run.smithi143:> uname -m 2020-03-27T13:52:35.053 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2020-03-27T13:52:35.054 INFO:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2020-03-27T13:52:35.120 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Linux" 2020-03-27T13:52:35.121 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="7 (Core)" 2020-03-27T13:52:35.121 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos" 2020-03-27T13:52:35.121 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora" 2020-03-27T13:52:35.121 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="7" 2020-03-27T13:52:35.121 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-03-27T13:52:35.121 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31" 2020-03-27T13:52:35.121 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-03-27T13:52:35.122 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://www.centos.org/" 2020-03-27T13:52:35.122 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-27T13:52:35.122 INFO:teuthology.orchestra.run.smithi143.stdout: 2020-03-27T13:52:35.122 INFO:teuthology.orchestra.run.smithi143.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-03-27T13:52:35.122 INFO:teuthology.orchestra.run.smithi143.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-03-27T13:52:35.122 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-27T13:52:35.122 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-03-27T13:52:35.122 INFO:teuthology.orchestra.run.smithi143.stdout: 2020-03-27T13:52:35.123 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2020-03-27T13:52:35.152 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-03-27T13:52:35.178 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-03-27T13:52:35.391 INFO:teuthology.task.internal:Checking for old test directory... 2020-03-27T13:52:35.391 INFO:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2020-03-27T13:52:35.398 INFO:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2020-03-27T13:52:35.485 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-03-27T13:52:35.722 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-03-27T13:52:35.723 INFO:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2020-03-27T13:52:35.730 INFO:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2020-03-27T13:52:35.837 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-03-27T13:52:35.897 INFO:teuthology.run_tasks:Running task kernel... 2020-03-27T13:52:35.916 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.5': {'sha1': 'distro', 'kdb': True}, 'osd.4': {'sha1': 'distro', 'kdb': True}, 'osd.7': {'sha1': 'distro', 'kdb': True}, 'osd.6': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'mgr.y': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}, 'client.1': {'sha1': 'distro', 'kdb': True}} 2020-03-27T13:52:35.916 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-03-27T13:52:35.916 INFO:teuthology.orchestra.run.smithi088:> uname -r 2020-03-27T13:52:36.010 INFO:teuthology.orchestra.run.smithi088.stdout:3.10.0-957.21.3.el7.x86_64 2020-03-27T13:52:36.011 INFO:teuthology.task.kernel:Running kernel on smithi088: 3.10.0-957.21.3.el7.x86_64 2020-03-27T13:52:36.011 INFO:teuthology.orchestra.run.smithi088:> sudo yum install -y kernel 2020-03-27T13:52:36.854 INFO:teuthology.orchestra.run.smithi088.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-27T13:52:36.969 INFO:teuthology.orchestra.run.smithi088.stdout:Determining fastest mirrors 2020-03-27T13:52:40.480 INFO:teuthology.orchestra.run.smithi088.stdout: * base: mirror.facebook.net 2020-03-27T13:52:40.481 INFO:teuthology.orchestra.run.smithi088.stdout: * epel: ftp.linux.ncsu.edu 2020-03-27T13:52:40.481 INFO:teuthology.orchestra.run.smithi088.stdout: * extras: mirror.ash.fastserv.com 2020-03-27T13:52:40.482 INFO:teuthology.orchestra.run.smithi088.stdout: * updates: mirror.facebook.net 2020-03-27T13:52:41.029 INFO:teuthology.orchestra.run.smithi088.stderr:http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/7a1a1c6bf6b4ff915216b87bda9c03a57b3a7955d583d3e21125fa732207ff76-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2020-03-27T13:52:41.029 INFO:teuthology.orchestra.run.smithi088.stderr:Trying other mirror. 2020-03-27T13:52:41.030 INFO:teuthology.orchestra.run.smithi088.stderr:To address this issue please refer to the below wiki article 2020-03-27T13:52:41.030 INFO:teuthology.orchestra.run.smithi088.stderr: 2020-03-27T13:52:41.030 INFO:teuthology.orchestra.run.smithi088.stderr:https://wiki.centos.org/yum-errors 2020-03-27T13:52:41.030 INFO:teuthology.orchestra.run.smithi088.stderr: 2020-03-27T13:52:41.031 INFO:teuthology.orchestra.run.smithi088.stderr:If above article doesn't help to resolve this issue please use https://bugs.centos.org/. 2020-03-27T13:52:41.031 INFO:teuthology.orchestra.run.smithi088.stderr: 2020-03-27T13:52:41.105 INFO:teuthology.orchestra.run.smithi088.stderr:http://mirror.oss.ou.edu/epel/7/x86_64/repodata/5cfb7e00842e2d2e0293ba5c4e7f5c1be59a9e09d8e76c3963492b9ea8b2d6fc-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found 2020-03-27T13:52:41.105 INFO:teuthology.orchestra.run.smithi088.stderr:Trying other mirror. 2020-03-27T13:52:46.482 INFO:teuthology.orchestra.run.smithi088.stdout:1 packages excluded due to repository priority protections 2020-03-27T13:52:47.025 INFO:teuthology.orchestra.run.smithi088.stdout:Resolving Dependencies 2020-03-27T13:52:47.025 INFO:teuthology.orchestra.run.smithi088.stdout:--> Running transaction check 2020-03-27T13:52:47.025 INFO:teuthology.orchestra.run.smithi088.stdout:---> Package kernel.x86_64 0:3.10.0-1062.18.1.el7 will be installed 2020-03-27T13:52:47.271 INFO:teuthology.orchestra.run.smithi088.stdout:--> Processing Dependency: linux-firmware >= 20190429-72 for package: kernel-3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:52:47.375 INFO:teuthology.orchestra.run.smithi088.stdout:--> Running transaction check 2020-03-27T13:52:47.375 INFO:teuthology.orchestra.run.smithi088.stdout:---> Package linux-firmware.noarch 0:20180911-69.git85c5d90.el7 will be updated 2020-03-27T13:52:47.389 INFO:teuthology.orchestra.run.smithi088.stdout:---> Package linux-firmware.noarch 0:20190429-72.gitddde598.el7 will be an update 2020-03-27T13:52:48.431 INFO:teuthology.orchestra.run.smithi088.stdout:--> Finished Dependency Resolution 2020-03-27T13:52:49.169 INFO:teuthology.orchestra.run.smithi088.stdout: 2020-03-27T13:52:49.170 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies Resolved 2020-03-27T13:52:49.173 INFO:teuthology.orchestra.run.smithi088.stdout: 2020-03-27T13:52:49.173 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2020-03-27T13:52:49.173 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2020-03-27T13:52:49.173 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2020-03-27T13:52:49.174 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2020-03-27T13:52:49.174 INFO:teuthology.orchestra.run.smithi088.stdout: kernel x86_64 3.10.0-1062.18.1.el7 updates 50 M 2020-03-27T13:52:49.174 INFO:teuthology.orchestra.run.smithi088.stdout:Updating for dependencies: 2020-03-27T13:52:49.174 INFO:teuthology.orchestra.run.smithi088.stdout: linux-firmware noarch 20190429-72.gitddde598.el7 base 73 M 2020-03-27T13:52:49.174 INFO:teuthology.orchestra.run.smithi088.stdout: 2020-03-27T13:52:49.174 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2020-03-27T13:52:49.174 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2020-03-27T13:52:49.174 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2020-03-27T13:52:49.175 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrade ( 1 Dependent package) 2020-03-27T13:52:49.175 INFO:teuthology.orchestra.run.smithi088.stdout: 2020-03-27T13:52:49.175 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 123 M 2020-03-27T13:52:49.175 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading packages: 2020-03-27T13:52:49.175 INFO:teuthology.orchestra.run.smithi088.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2020-03-27T13:52:51.411 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2020-03-27T13:52:51.411 INFO:teuthology.orchestra.run.smithi088.stdout:Total 55 MB/s | 123 MB 00:02 2020-03-27T13:52:51.486 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2020-03-27T13:52:51.669 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2020-03-27T13:52:52.652 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded 2020-03-27T13:52:52.652 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2020-03-27T13:53:03.279 INFO:teuthology.orchestra.run.smithi088.stdout: Updating : linux-firmware-20190429-72.gitddde598.el7.noarch 1/3 2020-03-27T13:53:06.079 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-3.10.0-1062.18.1.el7.x86_64 2/3 2020-03-27T13:53:35.866 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2020-03-27T13:53:36.013 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-3.10.0-1062.18.1.el7.x86_64 1/3 2020-03-27T13:53:36.099 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : linux-firmware-20190429-72.gitddde598.el7.noarch 2/3 2020-03-27T13:53:36.430 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2020-03-27T13:53:36.430 INFO:teuthology.orchestra.run.smithi088.stdout: 2020-03-27T13:53:36.430 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2020-03-27T13:53:36.430 INFO:teuthology.orchestra.run.smithi088.stdout: kernel.x86_64 0:3.10.0-1062.18.1.el7 2020-03-27T13:53:36.430 INFO:teuthology.orchestra.run.smithi088.stdout: 2020-03-27T13:53:36.430 INFO:teuthology.orchestra.run.smithi088.stdout:Dependency Updated: 2020-03-27T13:53:36.431 INFO:teuthology.orchestra.run.smithi088.stdout: linux-firmware.noarch 0:20190429-72.gitddde598.el7 2020-03-27T13:53:36.431 INFO:teuthology.orchestra.run.smithi088.stdout: 2020-03-27T13:53:36.431 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2020-03-27T13:53:36.568 INFO:teuthology.orchestra.run.smithi088:> rpm -q kernel --last | head -n 1 2020-03-27T13:53:37.002 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-3.10.0-1062.18.1.el7.x86_64 Fri 27 Mar 2020 01:53:03 PM UTC 2020-03-27T13:53:37.004 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:53:37.004 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-957.21.3.el7.x86_64 Expected: 3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:53:37.005 INFO:teuthology.orchestra.run.smithi143:> uname -r 2020-03-27T13:53:37.096 INFO:teuthology.orchestra.run.smithi143.stdout:3.10.0-957.21.3.el7.x86_64 2020-03-27T13:53:37.096 INFO:teuthology.task.kernel:Running kernel on smithi143: 3.10.0-957.21.3.el7.x86_64 2020-03-27T13:53:37.097 INFO:teuthology.orchestra.run.smithi143:> sudo yum install -y kernel 2020-03-27T13:53:38.011 INFO:teuthology.orchestra.run.smithi143.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-27T13:53:38.119 INFO:teuthology.orchestra.run.smithi143.stdout:Determining fastest mirrors 2020-03-27T13:53:41.541 INFO:teuthology.orchestra.run.smithi143.stdout: * base: mirror.facebook.net 2020-03-27T13:53:41.553 INFO:teuthology.orchestra.run.smithi143.stdout: * epel: ftp.linux.ncsu.edu 2020-03-27T13:53:41.553 INFO:teuthology.orchestra.run.smithi143.stdout: * extras: mirror.ash.fastserv.com 2020-03-27T13:53:41.554 INFO:teuthology.orchestra.run.smithi143.stdout: * updates: mirror.facebook.net 2020-03-27T13:53:42.006 INFO:teuthology.orchestra.run.smithi143.stderr:http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/7a1a1c6bf6b4ff915216b87bda9c03a57b3a7955d583d3e21125fa732207ff76-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2020-03-27T13:53:42.007 INFO:teuthology.orchestra.run.smithi143.stderr:Trying other mirror. 2020-03-27T13:53:42.007 INFO:teuthology.orchestra.run.smithi143.stderr:To address this issue please refer to the below wiki article 2020-03-27T13:53:42.007 INFO:teuthology.orchestra.run.smithi143.stderr: 2020-03-27T13:53:42.007 INFO:teuthology.orchestra.run.smithi143.stderr:https://wiki.centos.org/yum-errors 2020-03-27T13:53:42.007 INFO:teuthology.orchestra.run.smithi143.stderr: 2020-03-27T13:53:42.007 INFO:teuthology.orchestra.run.smithi143.stderr:If above article doesn't help to resolve this issue please use https://bugs.centos.org/. 2020-03-27T13:53:42.008 INFO:teuthology.orchestra.run.smithi143.stderr: 2020-03-27T13:53:42.107 INFO:teuthology.orchestra.run.smithi143.stderr:http://mirror.oss.ou.edu/epel/7/x86_64/repodata/5cfb7e00842e2d2e0293ba5c4e7f5c1be59a9e09d8e76c3963492b9ea8b2d6fc-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found 2020-03-27T13:53:42.107 INFO:teuthology.orchestra.run.smithi143.stderr:Trying other mirror. 2020-03-27T13:53:47.414 INFO:teuthology.orchestra.run.smithi143.stdout:1 packages excluded due to repository priority protections 2020-03-27T13:53:47.971 INFO:teuthology.orchestra.run.smithi143.stdout:Resolving Dependencies 2020-03-27T13:53:47.971 INFO:teuthology.orchestra.run.smithi143.stdout:--> Running transaction check 2020-03-27T13:53:47.972 INFO:teuthology.orchestra.run.smithi143.stdout:---> Package kernel.x86_64 0:3.10.0-1062.18.1.el7 will be installed 2020-03-27T13:53:48.191 INFO:teuthology.orchestra.run.smithi143.stdout:--> Processing Dependency: linux-firmware >= 20190429-72 for package: kernel-3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:53:48.299 INFO:teuthology.orchestra.run.smithi143.stdout:--> Running transaction check 2020-03-27T13:53:48.299 INFO:teuthology.orchestra.run.smithi143.stdout:---> Package linux-firmware.noarch 0:20180911-69.git85c5d90.el7 will be updated 2020-03-27T13:53:48.313 INFO:teuthology.orchestra.run.smithi143.stdout:---> Package linux-firmware.noarch 0:20190429-72.gitddde598.el7 will be an update 2020-03-27T13:53:49.374 INFO:teuthology.orchestra.run.smithi143.stdout:--> Finished Dependency Resolution 2020-03-27T13:53:50.053 INFO:teuthology.orchestra.run.smithi143.stdout: 2020-03-27T13:53:50.054 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies Resolved 2020-03-27T13:53:50.057 INFO:teuthology.orchestra.run.smithi143.stdout: 2020-03-27T13:53:50.058 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2020-03-27T13:53:50.058 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2020-03-27T13:53:50.058 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2020-03-27T13:53:50.058 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2020-03-27T13:53:50.058 INFO:teuthology.orchestra.run.smithi143.stdout: kernel x86_64 3.10.0-1062.18.1.el7 updates 50 M 2020-03-27T13:53:50.058 INFO:teuthology.orchestra.run.smithi143.stdout:Updating for dependencies: 2020-03-27T13:53:50.058 INFO:teuthology.orchestra.run.smithi143.stdout: linux-firmware noarch 20190429-72.gitddde598.el7 base 73 M 2020-03-27T13:53:50.059 INFO:teuthology.orchestra.run.smithi143.stdout: 2020-03-27T13:53:50.059 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2020-03-27T13:53:50.059 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2020-03-27T13:53:50.059 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2020-03-27T13:53:50.059 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrade ( 1 Dependent package) 2020-03-27T13:53:50.059 INFO:teuthology.orchestra.run.smithi143.stdout: 2020-03-27T13:53:50.060 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 123 M 2020-03-27T13:53:50.060 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading packages: 2020-03-27T13:53:50.060 INFO:teuthology.orchestra.run.smithi143.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2020-03-27T13:53:52.304 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2020-03-27T13:53:52.304 INFO:teuthology.orchestra.run.smithi143.stdout:Total 55 MB/s | 123 MB 00:02 2020-03-27T13:53:52.381 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2020-03-27T13:53:52.564 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2020-03-27T13:53:53.515 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded 2020-03-27T13:53:53.516 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2020-03-27T13:54:03.968 INFO:teuthology.orchestra.run.smithi143.stdout: Updating : linux-firmware-20190429-72.gitddde598.el7.noarch 1/3 2020-03-27T13:54:06.866 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-3.10.0-1062.18.1.el7.x86_64 2/3 2020-03-27T13:54:37.678 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2020-03-27T13:54:37.843 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-3.10.0-1062.18.1.el7.x86_64 1/3 2020-03-27T13:54:37.943 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : linux-firmware-20190429-72.gitddde598.el7.noarch 2/3 2020-03-27T13:54:38.265 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2020-03-27T13:54:38.265 INFO:teuthology.orchestra.run.smithi143.stdout: 2020-03-27T13:54:38.265 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2020-03-27T13:54:38.266 INFO:teuthology.orchestra.run.smithi143.stdout: kernel.x86_64 0:3.10.0-1062.18.1.el7 2020-03-27T13:54:38.266 INFO:teuthology.orchestra.run.smithi143.stdout: 2020-03-27T13:54:38.266 INFO:teuthology.orchestra.run.smithi143.stdout:Dependency Updated: 2020-03-27T13:54:38.266 INFO:teuthology.orchestra.run.smithi143.stdout: linux-firmware.noarch 0:20190429-72.gitddde598.el7 2020-03-27T13:54:38.266 INFO:teuthology.orchestra.run.smithi143.stdout: 2020-03-27T13:54:38.266 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2020-03-27T13:54:38.395 INFO:teuthology.orchestra.run.smithi143:> rpm -q kernel --last | head -n 1 2020-03-27T13:54:38.822 INFO:teuthology.orchestra.run.smithi143.stdout:kernel-3.10.0-1062.18.1.el7.x86_64 Fri 27 Mar 2020 01:54:03 PM UTC 2020-03-27T13:54:38.825 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:54:38.825 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-957.21.3.el7.x86_64 Expected: 3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:54:38.825 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-03-27T13:54:38.825 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-03-27T13:54:38.825 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-03-27T13:54:38.826 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2020-03-27T13:54:38.826 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2020-03-27T13:54:38.826 INFO:teuthology.orchestra.run.smithi088:> rpm -q kernel --last | head -n 1 2020-03-27T13:54:39.181 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-3.10.0-1062.18.1.el7.x86_64 Fri 27 Mar 2020 01:53:03 PM UTC 2020-03-27T13:54:39.183 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:54:39.183 INFO:teuthology.orchestra.run.smithi088:> sudo rpm -qi grub2-tools 2020-03-27T13:54:39.329 INFO:teuthology.orchestra.run.smithi088.stdout:Name : grub2-tools 2020-03-27T13:54:39.329 INFO:teuthology.orchestra.run.smithi088.stdout:Epoch : 1 2020-03-27T13:54:39.329 INFO:teuthology.orchestra.run.smithi088.stdout:Version : 2.02 2020-03-27T13:54:39.329 INFO:teuthology.orchestra.run.smithi088.stdout:Release : 0.65.el7.centos.2 2020-03-27T13:54:39.329 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: x86_64 2020-03-27T13:54:39.329 INFO:teuthology.orchestra.run.smithi088.stdout:Install Date: Wed 10 Jul 2019 06:31:35 PM UTC 2020-03-27T13:54:39.330 INFO:teuthology.orchestra.run.smithi088.stdout:Group : System Environment/Base 2020-03-27T13:54:39.330 INFO:teuthology.orchestra.run.smithi088.stdout:Size : 10075694 2020-03-27T13:54:39.330 INFO:teuthology.orchestra.run.smithi088.stdout:License : GPLv3+ 2020-03-27T13:54:39.330 INFO:teuthology.orchestra.run.smithi088.stdout:Signature : RSA/SHA256, Mon 23 Oct 2017 01:30:31 PM UTC, Key ID 24c6a8a7f4a80eb5 2020-03-27T13:54:39.330 INFO:teuthology.orchestra.run.smithi088.stdout:Source RPM : grub2-2.02-0.65.el7.centos.2.src.rpm 2020-03-27T13:54:39.330 INFO:teuthology.orchestra.run.smithi088.stdout:Build Date : Sat 21 Oct 2017 08:12:51 AM UTC 2020-03-27T13:54:39.330 INFO:teuthology.orchestra.run.smithi088.stdout:Build Host : kbuilder.dev.centos.org 2020-03-27T13:54:39.331 INFO:teuthology.orchestra.run.smithi088.stdout:Relocations : (not relocatable) 2020-03-27T13:54:39.331 INFO:teuthology.orchestra.run.smithi088.stdout:Packager : CentOS BuildSystem 2020-03-27T13:54:39.331 INFO:teuthology.orchestra.run.smithi088.stdout:Vendor : CentOS 2020-03-27T13:54:39.331 INFO:teuthology.orchestra.run.smithi088.stdout:URL : http://www.gnu.org/software/grub/ 2020-03-27T13:54:39.331 INFO:teuthology.orchestra.run.smithi088.stdout:Summary : Support tools for GRUB. 2020-03-27T13:54:39.331 INFO:teuthology.orchestra.run.smithi088.stdout:Description : 2020-03-27T13:54:39.331 INFO:teuthology.orchestra.run.smithi088.stdout: 2020-03-27T13:54:39.331 INFO:teuthology.orchestra.run.smithi088.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-03-27T13:54:39.332 INFO:teuthology.orchestra.run.smithi088.stdout:customizable bootloader with modular architecture. It supports a rich 2020-03-27T13:54:39.332 INFO:teuthology.orchestra.run.smithi088.stdout:variety of kernel formats, file systems, computer architectures and 2020-03-27T13:54:39.332 INFO:teuthology.orchestra.run.smithi088.stdout:hardware devices. 2020-03-27T13:54:39.332 INFO:teuthology.orchestra.run.smithi088.stdout: 2020-03-27T13:54:39.332 INFO:teuthology.orchestra.run.smithi088.stdout:This subpackage provides tools for support of all platforms. 2020-03-27T13:54:39.333 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-03-27T13:54:39.333 INFO:teuthology.task.kernel:Updating grub on smithi088 to boot 3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:54:39.334 INFO:teuthology.orchestra.run.smithi088:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-03-27T13:54:39.786 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2020-03-27T13:54:39.943 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:54:39.960 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img 2020-03-27T13:54:40.261 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.21.3.el7.x86_64 2020-03-27T13:54:40.274 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initramfs-3.10.0-957.21.3.el7.x86_64.img 2020-03-27T13:54:40.307 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-3.10.0-862.el7.x86_64 2020-03-27T13:54:40.330 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initramfs-3.10.0-862.el7.x86_64.img 2020-03-27T13:54:40.357 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-0-rescue-c960826f7edc464ebfd69d57481a61ea 2020-03-27T13:54:40.367 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initramfs-0-rescue-c960826f7edc464ebfd69d57481a61ea.img 2020-03-27T13:54:41.566 INFO:teuthology.orchestra.run.smithi088.stderr:done 2020-03-27T13:54:41.567 INFO:teuthology.orchestra.run.smithi088:> mktemp 2020-03-27T13:54:41.619 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.MNTg9ueSK2 2020-03-27T13:54:41.619 INFO:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MNTg9ueSK2 2020-03-27T13:54:41.673 INFO:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.MNTg9ueSK2 2020-03-27T13:54:41.789 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.MNTg9ueSK2 is 6KB 2020-03-27T13:54:41.840 INFO:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.MNTg9ueSK2 2020-03-27T13:54:41.885 INFO:teuthology.orchestra.run.smithi088:> sudo grub2-set-default 0 2020-03-27T13:54:42.097 INFO:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2020-03-27T13:54:42.101 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2020-03-27T13:54:42.101 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi143.front.sepia.ceph.com, path=None, version=distro) 2020-03-27T13:54:42.101 INFO:teuthology.orchestra.run.smithi143:> rpm -q kernel --last | head -n 1 2020-03-27T13:54:42.455 INFO:teuthology.orchestra.run.smithi143.stdout:kernel-3.10.0-1062.18.1.el7.x86_64 Fri 27 Mar 2020 01:54:03 PM UTC 2020-03-27T13:54:42.458 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:54:42.458 INFO:teuthology.orchestra.run.smithi143:> sudo rpm -qi grub2-tools 2020-03-27T13:54:42.601 INFO:teuthology.orchestra.run.smithi143.stdout:Name : grub2-tools 2020-03-27T13:54:42.601 INFO:teuthology.orchestra.run.smithi143.stdout:Epoch : 1 2020-03-27T13:54:42.601 INFO:teuthology.orchestra.run.smithi143.stdout:Version : 2.02 2020-03-27T13:54:42.601 INFO:teuthology.orchestra.run.smithi143.stdout:Release : 0.65.el7.centos.2 2020-03-27T13:54:42.602 INFO:teuthology.orchestra.run.smithi143.stdout:Architecture: x86_64 2020-03-27T13:54:42.602 INFO:teuthology.orchestra.run.smithi143.stdout:Install Date: Wed 10 Jul 2019 06:31:35 PM UTC 2020-03-27T13:54:42.602 INFO:teuthology.orchestra.run.smithi143.stdout:Group : System Environment/Base 2020-03-27T13:54:42.602 INFO:teuthology.orchestra.run.smithi143.stdout:Size : 10075694 2020-03-27T13:54:42.602 INFO:teuthology.orchestra.run.smithi143.stdout:License : GPLv3+ 2020-03-27T13:54:42.602 INFO:teuthology.orchestra.run.smithi143.stdout:Signature : RSA/SHA256, Mon 23 Oct 2017 01:30:31 PM UTC, Key ID 24c6a8a7f4a80eb5 2020-03-27T13:54:42.602 INFO:teuthology.orchestra.run.smithi143.stdout:Source RPM : grub2-2.02-0.65.el7.centos.2.src.rpm 2020-03-27T13:54:42.602 INFO:teuthology.orchestra.run.smithi143.stdout:Build Date : Sat 21 Oct 2017 08:12:51 AM UTC 2020-03-27T13:54:42.603 INFO:teuthology.orchestra.run.smithi143.stdout:Build Host : kbuilder.dev.centos.org 2020-03-27T13:54:42.603 INFO:teuthology.orchestra.run.smithi143.stdout:Relocations : (not relocatable) 2020-03-27T13:54:42.603 INFO:teuthology.orchestra.run.smithi143.stdout:Packager : CentOS BuildSystem 2020-03-27T13:54:42.603 INFO:teuthology.orchestra.run.smithi143.stdout:Vendor : CentOS 2020-03-27T13:54:42.603 INFO:teuthology.orchestra.run.smithi143.stdout:URL : http://www.gnu.org/software/grub/ 2020-03-27T13:54:42.603 INFO:teuthology.orchestra.run.smithi143.stdout:Summary : Support tools for GRUB. 2020-03-27T13:54:42.603 INFO:teuthology.orchestra.run.smithi143.stdout:Description : 2020-03-27T13:54:42.604 INFO:teuthology.orchestra.run.smithi143.stdout: 2020-03-27T13:54:42.604 INFO:teuthology.orchestra.run.smithi143.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-03-27T13:54:42.604 INFO:teuthology.orchestra.run.smithi143.stdout:customizable bootloader with modular architecture. It supports a rich 2020-03-27T13:54:42.604 INFO:teuthology.orchestra.run.smithi143.stdout:variety of kernel formats, file systems, computer architectures and 2020-03-27T13:54:42.604 INFO:teuthology.orchestra.run.smithi143.stdout:hardware devices. 2020-03-27T13:54:42.604 INFO:teuthology.orchestra.run.smithi143.stdout: 2020-03-27T13:54:42.604 INFO:teuthology.orchestra.run.smithi143.stdout:This subpackage provides tools for support of all platforms. 2020-03-27T13:54:42.605 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-03-27T13:54:42.605 INFO:teuthology.task.kernel:Updating grub on smithi143 to boot 3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:54:42.606 INFO:teuthology.orchestra.run.smithi143:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-03-27T13:54:43.078 INFO:teuthology.orchestra.run.smithi143.stderr:Generating grub configuration file ... 2020-03-27T13:54:43.196 INFO:teuthology.orchestra.run.smithi143.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:54:43.210 INFO:teuthology.orchestra.run.smithi143.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img 2020-03-27T13:54:43.465 INFO:teuthology.orchestra.run.smithi143.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.21.3.el7.x86_64 2020-03-27T13:54:43.466 INFO:teuthology.orchestra.run.smithi143.stderr:Found initrd image: /boot/initramfs-3.10.0-957.21.3.el7.x86_64.img 2020-03-27T13:54:43.490 INFO:teuthology.orchestra.run.smithi143.stderr:Found linux image: /boot/vmlinuz-3.10.0-862.el7.x86_64 2020-03-27T13:54:43.509 INFO:teuthology.orchestra.run.smithi143.stderr:Found initrd image: /boot/initramfs-3.10.0-862.el7.x86_64.img 2020-03-27T13:54:43.534 INFO:teuthology.orchestra.run.smithi143.stderr:Found linux image: /boot/vmlinuz-0-rescue-c960826f7edc464ebfd69d57481a61ea 2020-03-27T13:54:43.543 INFO:teuthology.orchestra.run.smithi143.stderr:Found initrd image: /boot/initramfs-0-rescue-c960826f7edc464ebfd69d57481a61ea.img 2020-03-27T13:54:44.695 INFO:teuthology.orchestra.run.smithi143.stderr:done 2020-03-27T13:54:44.697 INFO:teuthology.orchestra.run.smithi143:> mktemp 2020-03-27T13:54:44.750 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.VYdd0LyKdq 2020-03-27T13:54:44.750 INFO:teuthology.orchestra.run.smithi143:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VYdd0LyKdq 2020-03-27T13:54:44.810 INFO:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.VYdd0LyKdq 2020-03-27T13:54:44.919 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.VYdd0LyKdq is 6KB 2020-03-27T13:54:44.969 INFO:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.VYdd0LyKdq 2020-03-27T13:54:45.013 INFO:teuthology.orchestra.run.smithi143:> sudo grub2-set-default 0 2020-03-27T13:54:45.231 INFO:teuthology.orchestra.run.smithi143:> sudo shutdown -r now 2020-03-27T13:55:15.262 INFO:teuthology.misc:Re-opening connections... 2020-03-27T13:55:15.263 INFO:teuthology.misc:trying to connect to ubuntu@smithi088.front.sepia.ceph.com 2020-03-27T13:55:15.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-27T13:55:15.263 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi088.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:55:33.723 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2020-03-27T13:55:33.723 INFO:teuthology.misc:trying to connect to ubuntu@smithi143.front.sepia.ceph.com 2020-03-27T13:55:33.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-27T13:55:33.723 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi143.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:55:34.415 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-03-27T13:55:34.415 DEBUG:teuthology.task.console_log:Killing console logger for smithi143 2020-03-27T13:55:34.416 DEBUG:teuthology.task.console_log:Killing console logger for smithi088 2020-03-27T13:55:34.416 DEBUG:teuthology.task.console_log:Killing console logger for smithi143 2020-03-27T13:55:34.416 DEBUG:teuthology.task.console_log:Killing console logger for smithi088 2020-03-27T13:55:34.416 DEBUG:teuthology.exit:Finished running handlers