2020-03-27T13:42:53.934 INFO:root:teuthology version: 1.0.0-b3a6b1a 2020-03-27T13:42:53.935 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.1x4pjl.tmp -- --verbose --lock --description rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/fastclose.yaml objectstore/bluestore-bitmap.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{centos_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=2-m=1.yaml} --name kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi --block --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi/4894905 2020-03-27T13:42:53.977 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-27T13:42:54.045 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi/4894905 branch: nautilus-kefu2-testing-2020-03-27-1709 description: rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/fastclose.yaml objectstore/bluestore-bitmap.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{centos_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=2-m=1.yaml} email: tchaikov@gmail.com first_in_suite: false job_id: '4894905' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: centos os_version: '7.5' overrides: admin_socket: branch: nautilus-kefu2-testing-2020-03-27-1709 ceph: conf: global: ms inject socket failures: 2500 ms tcp read timeout: 5 osd_recovery_max_active: 10 osd_recovery_max_single_start: 10 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 2 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: nautilus-kefu2-testing-2020-03-27-1709 sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a owner: scheduled_kchai@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a suite: rados suite_branch: nautilus-kefu2-testing-2020-03-27-1709 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-1709/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a tasks: - install: null - ceph: null - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 4 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: crush-failure-domain: osd k: 2 m: 1 name: jerasure21profile plugin: jerasure technique: reed_sol_van objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 4000 write_append_excl: false teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.2394 2020-03-27T13:42:54.046 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-1709/qa; will attempt to use it 2020-03-27T13:42:54.046 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-1709/qa/tasks 2020-03-27T13:42:54.046 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-27T13:42:54.048 INFO:teuthology.task.internal:Checking packages... 2020-03-27T13:42:54.063 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'e9aaac6a13136ef54c1c1edf22f45ec35b28b19a' 2020-03-27T13:42:54.063 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-27T13:42:54.063 INFO:teuthology.packaging:ref: None 2020-03-27T13:42:54.063 INFO:teuthology.packaging:tag: None 2020-03-27T13:42:54.063 INFO:teuthology.packaging:branch: nautilus-kefu2-testing-2020-03-27-1709 2020-03-27T13:42:54.063 INFO:teuthology.packaging:sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a 2020-03-27T13:42:54.064 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=nautilus-kefu2-testing-2020-03-27-1709 2020-03-27T13:42:54.281 INFO:teuthology.task.internal:Found packages for ceph version 14.2.8-299.ge9aaac6 2020-03-27T13:42:54.282 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-27T13:42:54.297 INFO:teuthology.task.internal:no buildpackages task found 2020-03-27T13:42:54.297 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-27T13:42:54.310 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-27T13:42:54.310 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-27T13:42:54.358 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:43:04.376 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:43:14.395 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:43:24.409 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:43:34.429 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:43:44.452 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:43:54.472 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:44:04.492 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:44:14.511 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:44:24.531 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:44:34.549 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:44:44.568 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-27T13:44:54.586 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:45:04.613 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:45:14.631 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:45:24.649 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:45:34.665 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:45:44.681 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:45:54.697 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:46:04.715 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:46:14.729 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:46:24.750 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:46:34.764 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:46:44.784 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:46:54.805 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:47:04.827 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:47:14.845 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:47:24.862 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:47:34.878 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:47:44.892 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:47:54.917 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:48:04.934 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:48:14.949 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-27T13:48:24.966 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-27T13:48:34.981 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_kchai@teuthology', 'description': '/home/teuthworker/archive/kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi/4894905', 'machine_type': 'smithi'} 2020-03-27T13:48:34.998 DEBUG:teuthology.lock.ops:locked ubuntu@smithi161.front.sepia.ceph.com, ubuntu@smithi049.front.sepia.ceph.com 2020-03-27T13:48:35.055 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-27T13:48:35.248 INFO:teuthology.provision.fog.smithi161:Scheduling deploy of centos 7.5 2020-03-27T13:48:35.259 INFO:teuthology.provision.fog.smithi049:Scheduling deploy of centos 7.5 2020-03-27T13:48:35.576 INFO:teuthology.orchestra.console:Power off smithi161 2020-03-27T13:48:35.577 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-27T13:48:35.598 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:48:35.612 INFO:teuthology.orchestra.console:Power off smithi049 2020-03-27T13:48:35.613 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-27T13:48:35.633 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:48:39.622 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:48:39.656 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:48:43.738 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:48:43.768 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:48:43.956 INFO:teuthology.orchestra.console:Power off for smithi161 completed 2020-03-27T13:48:43.980 INFO:teuthology.orchestra.console:Power off for smithi049 completed 2020-03-27T13:48:44.057 INFO:teuthology.orchestra.console:Power on smithi161 2020-03-27T13:48:44.058 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-27T13:48:44.081 INFO:teuthology.orchestra.console:Power on smithi049 2020-03-27T13:48:44.081 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-27T13:48:44.089 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:48:44.102 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:48:48.113 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:48:48.123 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:48:48.335 INFO:teuthology.orchestra.console:Power on for smithi049 completed 2020-03-27T13:48:48.436 INFO:teuthology.provision.fog.smithi049:Waiting for deploy to finish 2020-03-27T13:48:52.229 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:48:52.438 INFO:teuthology.orchestra.console:Power on for smithi161 completed 2020-03-27T13:48:52.539 INFO:teuthology.provision.fog.smithi161:Waiting for deploy to finish 2020-03-27T13:51:19.452 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi049.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:51:28.527 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi049.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:51:37.609 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi049.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:51:38.592 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi161.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:51:46.663 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi049.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:51:47.655 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi161.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:51:55.721 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi049.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:51:56.710 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi161.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:04.782 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi049.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:05.766 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi161.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:13.869 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi049.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:14.822 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi161.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:23.881 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi161.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:32.938 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi161.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:38.218 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi049.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:41.994 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi161.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:46.033 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi049.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:52.043 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi049.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:52:52.300 INFO:teuthology.orchestra.run.smithi049:> hostname 2020-03-27T13:52:52.755 INFO:teuthology.orchestra.run.smithi049.stdout:smithi124 2020-03-27T13:52:52.755 INFO:teuthology.orchestra.run.smithi049:> grep smithi124 /etc/hosts 2020-03-27T13:52:52.799 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.15.124 smithi124.front.sepia.ceph.com smithi124 2020-03-27T13:52:52.799 INFO:teuthology.orchestra.run.smithi049:> sudo hostname smithi049 2020-03-27T13:52:52.923 INFO:teuthology.orchestra.run.smithi049:> sudo sed -i -e 's/smithi124/smithi049/g' /etc/hosts 2020-03-27T13:52:52.975 INFO:teuthology.orchestra.run.smithi049:> sudo sed -i -e 's/172.21.15.124/172.21.15.49/g' /etc/hosts 2020-03-27T13:52:53.018 INFO:teuthology.orchestra.run.smithi049:> sudo sed -i -e 's/smithi124/smithi049/g' /etc/hostname 2020-03-27T13:52:53.107 INFO:teuthology.orchestra.run.smithi049:> sudo hostname smithi049 2020-03-27T13:52:53.193 INFO:teuthology.provision.fog.smithi049:Deploy complete! 2020-03-27T13:52:53.195 DEBUG:teuthology.parallel:result is None 2020-03-27T13:53:13.616 INFO:teuthology.orchestra.run.smithi161:> hostname 2020-03-27T13:53:14.083 INFO:teuthology.orchestra.run.smithi161.stdout:smithi124 2020-03-27T13:53:14.084 INFO:teuthology.orchestra.run.smithi161:> grep smithi124 /etc/hosts 2020-03-27T13:53:14.125 INFO:teuthology.orchestra.run.smithi161.stdout:172.21.15.124 smithi124.front.sepia.ceph.com smithi124 2020-03-27T13:53:14.126 INFO:teuthology.orchestra.run.smithi161:> sudo hostname smithi161 2020-03-27T13:53:14.251 INFO:teuthology.orchestra.run.smithi161:> sudo sed -i -e 's/smithi124/smithi161/g' /etc/hosts 2020-03-27T13:53:14.302 INFO:teuthology.orchestra.run.smithi161:> sudo sed -i -e 's/172.21.15.124/172.21.15.161/g' /etc/hosts 2020-03-27T13:53:14.348 INFO:teuthology.orchestra.run.smithi161:> sudo sed -i -e 's/smithi124/smithi161/g' /etc/hostname 2020-03-27T13:53:14.394 INFO:teuthology.orchestra.run.smithi161:> sudo hostname smithi161 2020-03-27T13:53:14.439 INFO:teuthology.provision.fog.smithi161:Deploy complete! 2020-03-27T13:53:14.440 DEBUG:teuthology.parallel:result is None 2020-03-27T13:53:14.591 INFO:teuthology.lock.ops:Checking smithi049.front.sepia.ceph.com 2020-03-27T13:53:14.591 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-27T13:53:14.617 INFO:teuthology.lock.ops:Checking smithi161.front.sepia.ceph.com 2020-03-27T13:53:14.617 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-27T13:53:14.653 INFO:teuthology.lock.ops:Updating [smithi049.front.sepia.ceph.com]: set os type and version on server 2020-03-27T13:53:14.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-27T13:53:14.654 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi049.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:53:14.755 INFO:teuthology.orchestra.run.smithi049:> true 2020-03-27T13:53:15.137 INFO:teuthology.orchestra.run.smithi049:> uname -m 2020-03-27T13:53:15.217 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2020-03-27T13:53:15.218 INFO:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2020-03-27T13:53:15.285 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Linux" 2020-03-27T13:53:15.286 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="7 (Core)" 2020-03-27T13:53:15.286 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2020-03-27T13:53:15.286 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2020-03-27T13:53:15.286 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="7" 2020-03-27T13:53:15.286 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-03-27T13:53:15.286 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2020-03-27T13:53:15.286 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-03-27T13:53:15.287 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://www.centos.org/" 2020-03-27T13:53:15.287 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-27T13:53:15.287 INFO:teuthology.orchestra.run.smithi049.stdout: 2020-03-27T13:53:15.287 INFO:teuthology.orchestra.run.smithi049.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-03-27T13:53:15.287 INFO:teuthology.orchestra.run.smithi049.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-03-27T13:53:15.287 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-27T13:53:15.287 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-03-27T13:53:15.288 INFO:teuthology.orchestra.run.smithi049.stdout: 2020-03-27T13:53:15.288 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2020-03-27T13:53:15.302 INFO:teuthology.lock.ops:Updating [smithi161.front.sepia.ceph.com]: set os type and version on server 2020-03-27T13:53:15.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-27T13:53:15.302 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi161.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:53:15.371 INFO:teuthology.orchestra.run.smithi161:> true 2020-03-27T13:53:15.627 INFO:teuthology.orchestra.run.smithi161:> uname -m 2020-03-27T13:53:15.670 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2020-03-27T13:53:15.670 INFO:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2020-03-27T13:53:15.707 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Linux" 2020-03-27T13:53:15.708 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="7 (Core)" 2020-03-27T13:53:15.708 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos" 2020-03-27T13:53:15.708 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora" 2020-03-27T13:53:15.708 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="7" 2020-03-27T13:53:15.708 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-03-27T13:53:15.708 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2020-03-27T13:53:15.709 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-03-27T13:53:15.709 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://www.centos.org/" 2020-03-27T13:53:15.709 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-27T13:53:15.709 INFO:teuthology.orchestra.run.smithi161.stdout: 2020-03-27T13:53:15.709 INFO:teuthology.orchestra.run.smithi161.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-03-27T13:53:15.709 INFO:teuthology.orchestra.run.smithi161.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-03-27T13:53:15.709 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-27T13:53:15.710 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-03-27T13:53:15.710 INFO:teuthology.orchestra.run.smithi161.stdout: 2020-03-27T13:53:15.710 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2020-03-27T13:53:15.729 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-03-27T13:53:15.766 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi049.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDG67RSuKvhNmzW43pAX6UTx0quIyKKjbb3VTTF5zIVS2rtrLgmFAscM2IfiIKaN/iR/YTF1+VqzspVh3PqXnwBAeKMw1vEdxclgP5Rw1UDVke45WKPdo0RLmv69thxmVvmBRCw8ysTZiKYf2HAJSJ99krBgfEiOcVe75HCJyRtEO2X1qyB6EIlGwD2LkR0dbF1UXHRtF/vno6py+EI8737hdtcpBQuT1I818Xnb+SxzD0GAkWq16BQeIzJQ2JZ9Eeif59BLKkfTYTy+QawXnyYz0WtWI+IR0dGIm/7sjjH/ZN08Fh7dPe5XBJL8Aezijah3wCH3eC/twzQc5/r8jxd smithi161.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCbMTk2qhzAHrdYBzUtxRWmNFLlG2ljbYwpzu8yo+uMnObSbjYj/FiDfZLzV4glD+5mYk1dXvzMoUS3eCbBeR+QyymYy6ftWIFIyBdGi5JQolmvH+Kg0ubiiM1oKNH5PGakK+iWHbfqNK6s6J+6A+6KhtlFhIq55ArCJ9Pbsf8Hf+IsIc+YYUDp3BpolM+ExRQAuJ5tZ+SaonnetvFpm1fXbAZt3uxDYj5qsYZ/aLH6dZWYibRxdrMrYv+UrhIFRQDDjrs+6kGKbiYaSYlFrbTTHl2ZTn5WBQEaXDe80DuRXVqSZzH7PNh4cXO1D2I2jgtqusGGUkASmOH1Itc1rUph 2020-03-27T13:53:15.767 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-27T13:53:15.811 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-03-27T13:53:15.814 INFO:teuthology.task.internal:Saving configuration 2020-03-27T13:53:15.831 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-03-27T13:53:15.841 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-03-27T13:53:15.858 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-27 13:48:34.983984', u'locked_by': u'scheduled_kchai@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi049.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBESPAKOT4Xp6q83R6wQhNdpDFTWYhGAKrDb6EyYnP5jI86wcgbQkhNrGO3u7nad1Bd6Sx53KWuo5xqhhWY9LIco=', u'description': u'/home/teuthworker/archive/kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi/4894905'} 2020-03-27T13:53:15.874 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-27 13:48:34.984004', u'locked_by': u'scheduled_kchai@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi161.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLEdmLjLQaUd88MwxtVABF0LCSkav0IcNpPASuT1bS3nq5CMroSqBZBrmbtPa+vpEM3FEN0Z5Vfj9oXG79tWQ3c=', u'description': u'/home/teuthworker/archive/kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi/4894905'} 2020-03-27T13:53:15.874 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-03-27T13:53:15.881 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2020-03-27T13:53:15.881 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2020-03-27T13:53:15.881 INFO:teuthology.run_tasks:Running task console_log... 2020-03-27T13:53:16.020 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-27T13:53:16.021 INFO:teuthology.run_tasks:Running task internal.connect... 2020-03-27T13:53:16.028 INFO:teuthology.task.internal:Opening connections... 2020-03-27T13:53:16.028 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2020-03-27T13:53:16.029 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi161.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:53:16.098 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi049.front.sepia.ceph.com 2020-03-27T13:53:16.098 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi049.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:53:16.205 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-03-27T13:53:16.213 INFO:teuthology.orchestra.run.smithi161:> uname -m 2020-03-27T13:53:16.302 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2020-03-27T13:53:16.302 INFO:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2020-03-27T13:53:16.344 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Linux" 2020-03-27T13:53:16.344 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="7 (Core)" 2020-03-27T13:53:16.344 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos" 2020-03-27T13:53:16.344 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora" 2020-03-27T13:53:16.344 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="7" 2020-03-27T13:53:16.344 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-03-27T13:53:16.345 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2020-03-27T13:53:16.345 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-03-27T13:53:16.345 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://www.centos.org/" 2020-03-27T13:53:16.345 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-27T13:53:16.345 INFO:teuthology.orchestra.run.smithi161.stdout: 2020-03-27T13:53:16.345 INFO:teuthology.orchestra.run.smithi161.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-03-27T13:53:16.345 INFO:teuthology.orchestra.run.smithi161.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-03-27T13:53:16.345 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-27T13:53:16.346 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-03-27T13:53:16.346 INFO:teuthology.orchestra.run.smithi161.stdout: 2020-03-27T13:53:16.346 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2020-03-27T13:53:16.359 INFO:teuthology.orchestra.run.smithi049:> uname -m 2020-03-27T13:53:16.546 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2020-03-27T13:53:16.546 INFO:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2020-03-27T13:53:16.615 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Linux" 2020-03-27T13:53:16.615 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="7 (Core)" 2020-03-27T13:53:16.615 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2020-03-27T13:53:16.616 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2020-03-27T13:53:16.616 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="7" 2020-03-27T13:53:16.616 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-03-27T13:53:16.616 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2020-03-27T13:53:16.616 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-03-27T13:53:16.616 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://www.centos.org/" 2020-03-27T13:53:16.616 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-27T13:53:16.617 INFO:teuthology.orchestra.run.smithi049.stdout: 2020-03-27T13:53:16.617 INFO:teuthology.orchestra.run.smithi049.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-03-27T13:53:16.617 INFO:teuthology.orchestra.run.smithi049.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-03-27T13:53:16.617 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-27T13:53:16.617 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-03-27T13:53:16.617 INFO:teuthology.orchestra.run.smithi049.stdout: 2020-03-27T13:53:16.617 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2020-03-27T13:53:16.631 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-03-27T13:53:16.662 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-03-27T13:53:16.671 INFO:teuthology.task.internal:Checking for old test directory... 2020-03-27T13:53:16.671 INFO:teuthology.orchestra.run.smithi049:> test '!' -e /home/ubuntu/cephtest 2020-03-27T13:53:16.675 INFO:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2020-03-27T13:53:16.755 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-03-27T13:53:16.762 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-03-27T13:53:16.763 INFO:teuthology.orchestra.run.smithi049:> test -z $(ls -A /var/lib/ceph) 2020-03-27T13:53:16.766 INFO:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2020-03-27T13:53:16.851 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-03-27T13:53:16.887 INFO:teuthology.run_tasks:Running task kernel... 2020-03-27T13:53:16.908 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.1': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.5': {'sha1': 'distro', 'kdb': True}, 'osd.4': {'sha1': 'distro', 'kdb': True}, 'osd.7': {'sha1': 'distro', 'kdb': True}, 'osd.6': {'sha1': 'distro', 'kdb': True}, 'mgr.y': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}} 2020-03-27T13:53:16.908 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-03-27T13:53:16.909 INFO:teuthology.orchestra.run.smithi049:> uname -r 2020-03-27T13:53:16.993 INFO:teuthology.orchestra.run.smithi049.stdout:3.10.0-957.21.3.el7.x86_64 2020-03-27T13:53:16.993 INFO:teuthology.task.kernel:Running kernel on smithi049: 3.10.0-957.21.3.el7.x86_64 2020-03-27T13:53:16.993 INFO:teuthology.orchestra.run.smithi049:> sudo yum install -y kernel 2020-03-27T13:53:17.927 INFO:teuthology.orchestra.run.smithi049.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-27T13:53:18.041 INFO:teuthology.orchestra.run.smithi049.stdout:Determining fastest mirrors 2020-03-27T13:53:21.493 INFO:teuthology.orchestra.run.smithi049.stdout: * base: mirror.cs.vt.edu 2020-03-27T13:53:21.494 INFO:teuthology.orchestra.run.smithi049.stdout: * epel: ftp.linux.ncsu.edu 2020-03-27T13:53:21.494 INFO:teuthology.orchestra.run.smithi049.stdout: * extras: mirror.ash.fastserv.com 2020-03-27T13:53:21.495 INFO:teuthology.orchestra.run.smithi049.stdout: * updates: repos.mia.quadranet.com 2020-03-27T13:53:22.110 INFO:teuthology.orchestra.run.smithi049.stderr:http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/7a1a1c6bf6b4ff915216b87bda9c03a57b3a7955d583d3e21125fa732207ff76-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2020-03-27T13:53:22.110 INFO:teuthology.orchestra.run.smithi049.stderr:Trying other mirror. 2020-03-27T13:53:22.110 INFO:teuthology.orchestra.run.smithi049.stderr:To address this issue please refer to the below wiki article 2020-03-27T13:53:22.111 INFO:teuthology.orchestra.run.smithi049.stderr: 2020-03-27T13:53:22.111 INFO:teuthology.orchestra.run.smithi049.stderr:https://wiki.centos.org/yum-errors 2020-03-27T13:53:22.111 INFO:teuthology.orchestra.run.smithi049.stderr: 2020-03-27T13:53:22.111 INFO:teuthology.orchestra.run.smithi049.stderr:If above article doesn't help to resolve this issue please use https://bugs.centos.org/. 2020-03-27T13:53:22.111 INFO:teuthology.orchestra.run.smithi049.stderr: 2020-03-27T13:53:22.199 INFO:teuthology.orchestra.run.smithi049.stderr:http://mirror.oss.ou.edu/epel/7/x86_64/repodata/5cfb7e00842e2d2e0293ba5c4e7f5c1be59a9e09d8e76c3963492b9ea8b2d6fc-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found 2020-03-27T13:53:22.200 INFO:teuthology.orchestra.run.smithi049.stderr:Trying other mirror. 2020-03-27T13:53:26.820 INFO:teuthology.orchestra.run.smithi049.stdout:1 packages excluded due to repository priority protections 2020-03-27T13:53:27.361 INFO:teuthology.orchestra.run.smithi049.stdout:Resolving Dependencies 2020-03-27T13:53:27.362 INFO:teuthology.orchestra.run.smithi049.stdout:--> Running transaction check 2020-03-27T13:53:27.362 INFO:teuthology.orchestra.run.smithi049.stdout:---> Package kernel.x86_64 0:3.10.0-1062.18.1.el7 will be installed 2020-03-27T13:53:27.585 INFO:teuthology.orchestra.run.smithi049.stdout:--> Processing Dependency: linux-firmware >= 20190429-72 for package: kernel-3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:53:27.697 INFO:teuthology.orchestra.run.smithi049.stdout:--> Running transaction check 2020-03-27T13:53:27.697 INFO:teuthology.orchestra.run.smithi049.stdout:---> Package linux-firmware.noarch 0:20180911-69.git85c5d90.el7 will be updated 2020-03-27T13:53:27.708 INFO:teuthology.orchestra.run.smithi049.stdout:---> Package linux-firmware.noarch 0:20190429-72.gitddde598.el7 will be an update 2020-03-27T13:53:28.727 INFO:teuthology.orchestra.run.smithi049.stdout:--> Finished Dependency Resolution 2020-03-27T13:53:29.440 INFO:teuthology.orchestra.run.smithi049.stdout: 2020-03-27T13:53:29.440 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies Resolved 2020-03-27T13:53:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: 2020-03-27T13:53:29.455 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2020-03-27T13:53:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2020-03-27T13:53:29.455 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2020-03-27T13:53:29.455 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2020-03-27T13:53:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: kernel x86_64 3.10.0-1062.18.1.el7 updates 50 M 2020-03-27T13:53:29.456 INFO:teuthology.orchestra.run.smithi049.stdout:Updating for dependencies: 2020-03-27T13:53:29.456 INFO:teuthology.orchestra.run.smithi049.stdout: linux-firmware noarch 20190429-72.gitddde598.el7 base 73 M 2020-03-27T13:53:29.456 INFO:teuthology.orchestra.run.smithi049.stdout: 2020-03-27T13:53:29.456 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2020-03-27T13:53:29.456 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2020-03-27T13:53:29.456 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2020-03-27T13:53:29.456 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrade ( 1 Dependent package) 2020-03-27T13:53:29.457 INFO:teuthology.orchestra.run.smithi049.stdout: 2020-03-27T13:53:29.457 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 123 M 2020-03-27T13:53:29.457 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading packages: 2020-03-27T13:53:29.457 INFO:teuthology.orchestra.run.smithi049.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2020-03-27T13:53:37.993 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2020-03-27T13:53:37.993 INFO:teuthology.orchestra.run.smithi049.stdout:Total 14 MB/s | 123 MB 00:08 2020-03-27T13:53:38.071 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2020-03-27T13:53:38.256 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2020-03-27T13:53:39.132 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded 2020-03-27T13:53:39.133 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2020-03-27T13:53:50.290 INFO:teuthology.orchestra.run.smithi049.stdout: Updating : linux-firmware-20190429-72.gitddde598.el7.noarch 1/3 2020-03-27T13:53:52.972 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-3.10.0-1062.18.1.el7.x86_64 2/3 2020-03-27T13:54:23.982 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2020-03-27T13:54:24.138 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-3.10.0-1062.18.1.el7.x86_64 1/3 2020-03-27T13:54:24.255 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : linux-firmware-20190429-72.gitddde598.el7.noarch 2/3 2020-03-27T13:54:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2020-03-27T13:54:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: 2020-03-27T13:54:24.581 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2020-03-27T13:54:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: kernel.x86_64 0:3.10.0-1062.18.1.el7 2020-03-27T13:54:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: 2020-03-27T13:54:24.581 INFO:teuthology.orchestra.run.smithi049.stdout:Dependency Updated: 2020-03-27T13:54:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: linux-firmware.noarch 0:20190429-72.gitddde598.el7 2020-03-27T13:54:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: 2020-03-27T13:54:24.582 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2020-03-27T13:54:24.713 INFO:teuthology.orchestra.run.smithi049:> rpm -q kernel --last | head -n 1 2020-03-27T13:54:25.011 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-3.10.0-1062.18.1.el7.x86_64 Fri 27 Mar 2020 01:53:50 PM UTC 2020-03-27T13:54:25.013 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:54:25.013 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-957.21.3.el7.x86_64 Expected: 3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:54:25.013 INFO:teuthology.orchestra.run.smithi161:> uname -r 2020-03-27T13:54:25.138 INFO:teuthology.orchestra.run.smithi161.stdout:3.10.0-957.21.3.el7.x86_64 2020-03-27T13:54:25.138 INFO:teuthology.task.kernel:Running kernel on smithi161: 3.10.0-957.21.3.el7.x86_64 2020-03-27T13:54:25.138 INFO:teuthology.orchestra.run.smithi161:> sudo yum install -y kernel 2020-03-27T13:54:26.030 INFO:teuthology.orchestra.run.smithi161.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-27T13:54:26.161 INFO:teuthology.orchestra.run.smithi161.stdout:Determining fastest mirrors 2020-03-27T13:54:29.473 INFO:teuthology.orchestra.run.smithi161.stdout: * base: mirror.facebook.net 2020-03-27T13:54:29.478 INFO:teuthology.orchestra.run.smithi161.stdout: * epel: ftp.linux.ncsu.edu 2020-03-27T13:54:29.479 INFO:teuthology.orchestra.run.smithi161.stdout: * extras: mirror.ash.fastserv.com 2020-03-27T13:54:29.479 INFO:teuthology.orchestra.run.smithi161.stdout: * updates: mirror.facebook.net 2020-03-27T13:54:29.918 INFO:teuthology.orchestra.run.smithi161.stderr:http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/7a1a1c6bf6b4ff915216b87bda9c03a57b3a7955d583d3e21125fa732207ff76-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2020-03-27T13:54:29.919 INFO:teuthology.orchestra.run.smithi161.stderr:Trying other mirror. 2020-03-27T13:54:29.919 INFO:teuthology.orchestra.run.smithi161.stderr:To address this issue please refer to the below wiki article 2020-03-27T13:54:29.919 INFO:teuthology.orchestra.run.smithi161.stderr: 2020-03-27T13:54:29.919 INFO:teuthology.orchestra.run.smithi161.stderr:https://wiki.centos.org/yum-errors 2020-03-27T13:54:29.920 INFO:teuthology.orchestra.run.smithi161.stderr: 2020-03-27T13:54:29.920 INFO:teuthology.orchestra.run.smithi161.stderr:If above article doesn't help to resolve this issue please use https://bugs.centos.org/. 2020-03-27T13:54:29.920 INFO:teuthology.orchestra.run.smithi161.stderr: 2020-03-27T13:54:30.021 INFO:teuthology.orchestra.run.smithi161.stderr:http://mirror.oss.ou.edu/epel/7/x86_64/repodata/5cfb7e00842e2d2e0293ba5c4e7f5c1be59a9e09d8e76c3963492b9ea8b2d6fc-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found 2020-03-27T13:54:30.021 INFO:teuthology.orchestra.run.smithi161.stderr:Trying other mirror. 2020-03-27T13:54:38.325 INFO:teuthology.orchestra.run.smithi161.stdout:1 packages excluded due to repository priority protections 2020-03-27T13:54:38.873 INFO:teuthology.orchestra.run.smithi161.stdout:Resolving Dependencies 2020-03-27T13:54:38.874 INFO:teuthology.orchestra.run.smithi161.stdout:--> Running transaction check 2020-03-27T13:54:38.874 INFO:teuthology.orchestra.run.smithi161.stdout:---> Package kernel.x86_64 0:3.10.0-1062.18.1.el7 will be installed 2020-03-27T13:54:39.099 INFO:teuthology.orchestra.run.smithi161.stdout:--> Processing Dependency: linux-firmware >= 20190429-72 for package: kernel-3.10.0-1062.18.1.el7.x86_64 2020-03-27T13:54:39.223 INFO:teuthology.orchestra.run.smithi161.stdout:--> Running transaction check 2020-03-27T13:54:39.224 INFO:teuthology.orchestra.run.smithi161.stdout:---> Package linux-firmware.noarch 0:20180911-69.git85c5d90.el7 will be updated 2020-03-27T13:54:39.239 INFO:teuthology.orchestra.run.smithi161.stdout:---> Package linux-firmware.noarch 0:20190429-72.gitddde598.el7 will be an update 2020-03-27T13:54:40.315 INFO:teuthology.orchestra.run.smithi161.stdout:--> Finished Dependency Resolution 2020-03-27T13:54:41.001 INFO:teuthology.orchestra.run.smithi161.stdout: 2020-03-27T13:54:41.001 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies Resolved 2020-03-27T13:54:41.014 INFO:teuthology.orchestra.run.smithi161.stdout: 2020-03-27T13:54:41.014 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2020-03-27T13:54:41.014 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2020-03-27T13:54:41.015 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2020-03-27T13:54:41.015 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2020-03-27T13:54:41.015 INFO:teuthology.orchestra.run.smithi161.stdout: kernel x86_64 3.10.0-1062.18.1.el7 updates 50 M 2020-03-27T13:54:41.015 INFO:teuthology.orchestra.run.smithi161.stdout:Updating for dependencies: 2020-03-27T13:54:41.015 INFO:teuthology.orchestra.run.smithi161.stdout: linux-firmware noarch 20190429-72.gitddde598.el7 base 73 M 2020-03-27T13:54:41.015 INFO:teuthology.orchestra.run.smithi161.stdout: 2020-03-27T13:54:41.015 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2020-03-27T13:54:41.015 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2020-03-27T13:54:41.016 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2020-03-27T13:54:41.016 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrade ( 1 Dependent package) 2020-03-27T13:54:41.016 INFO:teuthology.orchestra.run.smithi161.stdout: 2020-03-27T13:54:41.016 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 123 M 2020-03-27T13:54:41.016 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading packages: 2020-03-27T13:54:41.017 INFO:teuthology.orchestra.run.smithi161.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2020-03-27T13:54:49.529 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2020-03-27T13:54:49.529 INFO:teuthology.orchestra.run.smithi161.stdout:Total 14 MB/s | 123 MB 00:08 2020-03-27T13:54:49.601 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2020-03-27T13:54:49.795 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2020-03-27T13:54:50.862 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded 2020-03-27T13:54:50.863 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2020-03-27T13:55:01.613 INFO:teuthology.orchestra.run.smithi161.stdout: Updating : linux-firmware-20190429-72.gitddde598.el7.noarch 1/3 2020-03-27T13:55:04.592 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-3.10.0-1062.18.1.el7.x86_64 2/3 2020-03-27T13:55:34.509 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-03-27T13:55:34.509 DEBUG:teuthology.task.console_log:Killing console logger for smithi161 2020-03-27T13:55:34.510 DEBUG:teuthology.task.console_log:Killing console logger for smithi049 2020-03-27T13:55:34.510 DEBUG:teuthology.task.console_log:Killing console logger for smithi161 2020-03-27T13:55:34.511 DEBUG:teuthology.task.console_log:Killing console logger for smithi049 2020-03-27T13:55:34.511 DEBUG:teuthology.exit:Finished running handlers