2019-11-04T19:03:56.742 INFO:root:teuthology version: 1.0.0-5052ff0 2019-11-04T19:03:56.743 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.re4pjN.tmp -- --verbose --lock --description rgw/tempest/{clusters/fixed-1.yaml distro/supported-random-distro$/{centos_7.yaml} frontend/civetweb.yaml tasks/rgw_tempest.yaml} --name dang-2019-11-04_07:46:57-rgw-wip-dang-zipper-3-distro-basic-smithi --block --owner scheduled_dang@sidious.arb.redhat.com --archive /home/teuthworker/archive/dang-2019-11-04_07:46:57-rgw-wip-dang-zipper-3-distro-basic-smithi/4472344 2019-11-04T19:03:56.788 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-11-04T19:03:56.844 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dang-2019-11-04_07:46:57-rgw-wip-dang-zipper-3-distro-basic-smithi/4472344 branch: wip-dang-zipper-3 description: rgw/tempest/{clusters/fixed-1.yaml distro/supported-random-distro$/{centos_7.yaml} frontend/civetweb.yaml tasks/rgw_tempest.yaml} email: null first_in_suite: false job_id: '4472344' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: dang-2019-11-04_07:46:57-rgw-wip-dang-zipper-3-distro-basic-smithi nuke-on-error: true os_type: centos os_version: '7.6' overrides: admin_socket: branch: wip-dang-zipper-3 ceph: conf: client: rgw keystone accepted admin roles: admin rgw keystone accepted roles: admin,Member rgw keystone admin token: ADMIN rgw keystone implicit tenants: true rgw swift account in url: true rgw swift enforce content length: true rgw swift versioning enabled: true global: osd_max_pg_log_entries: 10 osd_min_pg_log_entries: 10 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 osd shutdown pgref assert: true log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 2baa72b367b929aed64df5fc7c9bca889f3b7e90 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: sha1: 2baa72b367b929aed64df5fc7c9bca889f3b7e90 rgw: frontend: civetweb workunit: branch: wip-dang-zipper-3 sha1: 2baa72b367b929aed64df5fc7c9bca889f3b7e90 owner: scheduled_dang@sidious.arb.redhat.com priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 2baa72b367b929aed64df5fc7c9bca889f3b7e90 suite: rgw suite_branch: wip-dang-zipper-3 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-zipper-3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 2baa72b367b929aed64df5fc7c9bca889f3b7e90 tasks: - install: null - ceph: null - tox: - client.0 - keystone: client.0: force-branch: master role-mappings: - name: admin project: admin user: admin roles: - name: admin - name: Member services: - description: Keystone Identity Service name: keystone type: identity - description: Swift Service name: swift type: object-store sha1: 12.0.0.0b2 tenants: - description: Admin Tenant name: admin users: - name: admin password: ADMIN project: admin - rgw: client.0: frontend_prefix: /swift use-keystone-role: client.0 - tempest: client.0: auth: admin_domain_name: Default admin_password: ADMIN admin_project_name: admin admin_username: admin blacklist: - .*test_list_containers_reverse_order.* - .*test_list_container_contents_with_end_marker.* - .*test_delete_non_empty_container.* - .*test_container_synchronization.* - .*test_get_object_after_expiration_time.* - .*test_create_object_with_transfer_encoding.* force-branch: master identity: admin_role: admin uri: http://{keystone_public_host}:{keystone_public_port}/v2.0/ uri_v3: http://{keystone_public_host}:{keystone_public_port}/v3/ object-storage: reseller_admin_role: admin object-storage-feature-enabled: container_sync: false discoverability: false sha1: d3fa46495a78160989120ba39793f7ba2e22d81c use-keystone-role: client.0 teuthology_branch: master tube: smithi verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.15246 2019-11-04T19:03:56.844 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-zipper-3/qa; will attempt to use it 2019-11-04T19:03:56.845 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-zipper-3/qa/tasks 2019-11-04T19:03:56.845 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-11-04T19:03:56.846 INFO:teuthology.task.internal:Checking packages... 2019-11-04T19:03:56.873 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '2baa72b367b929aed64df5fc7c9bca889f3b7e90' 2019-11-04T19:03:56.874 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-11-04T19:03:56.874 INFO:teuthology.packaging:ref: None 2019-11-04T19:03:56.874 INFO:teuthology.packaging:tag: None 2019-11-04T19:03:56.874 INFO:teuthology.packaging:branch: wip-dang-zipper-3 2019-11-04T19:03:56.874 INFO:teuthology.packaging:sha1: 2baa72b367b929aed64df5fc7c9bca889f3b7e90 2019-11-04T19:03:56.875 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-dang-zipper-3 2019-11-04T19:03:57.045 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-6611.g2baa72b 2019-11-04T19:03:57.046 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-11-04T19:03:57.056 INFO:teuthology.task.internal:no buildpackages task found 2019-11-04T19:03:57.056 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-11-04T19:03:57.126 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-11-04T19:03:57.127 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-11-04T19:03:57.172 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_dang@sidious.arb.redhat.com', 'description': '/home/teuthworker/archive/dang-2019-11-04_07:46:57-rgw-wip-dang-zipper-3-distro-basic-smithi/4472344', 'machine_type': 'smithi'} 2019-11-04T19:03:57.186 DEBUG:teuthology.lock.ops:locked ubuntu@smithi070.front.sepia.ceph.com 2019-11-04T19:03:57.271 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-11-04T19:03:57.502 INFO:teuthology.provision.fog.smithi070:Scheduling deploy of centos 7.6 2019-11-04T19:03:57.849 INFO:teuthology.orchestra.console:Power off smithi070 2019-11-04T19:03:57.850 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-11-04T19:03:57.901 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-04T19:04:01.956 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-04T19:04:06.088 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-04T19:04:06.317 INFO:teuthology.orchestra.console:Power off for smithi070 completed 2019-11-04T19:04:06.417 INFO:teuthology.orchestra.console:Power on smithi070 2019-11-04T19:04:06.418 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-11-04T19:04:06.458 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-04T19:04:10.501 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-04T19:04:14.633 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-04T19:04:14.972 INFO:teuthology.orchestra.console:Power on for smithi070 completed 2019-11-04T19:04:15.073 INFO:teuthology.provision.fog.smithi070:Waiting for deploy to finish 2019-11-04T19:07:01.486 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi070.front.sepia.ceph.com', 'timeout': 60} 2019-11-04T19:07:10.541 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi070.front.sepia.ceph.com', 'timeout': 60} 2019-11-04T19:07:19.597 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi070.front.sepia.ceph.com', 'timeout': 60} 2019-11-04T19:07:28.653 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi070.front.sepia.ceph.com', 'timeout': 60} 2019-11-04T19:07:37.713 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi070.front.sepia.ceph.com', 'timeout': 60} 2019-11-04T19:07:46.801 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi070.front.sepia.ceph.com', 'timeout': 60} 2019-11-04T19:07:55.886 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi070.front.sepia.ceph.com', 'timeout': 60} 2019-11-04T19:08:04.941 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi070.front.sepia.ceph.com', 'timeout': 60} 2019-11-04T19:08:14.029 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi070.front.sepia.ceph.com', 'timeout': 60} 2019-11-04T19:08:45.896 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:08:45.896 INFO:teuthology.orchestra.run.smithi070:> hostname 2019-11-04T19:08:46.424 INFO:teuthology.orchestra.run.smithi070.stdout:smithi141 2019-11-04T19:08:46.425 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:08:46.425 INFO:teuthology.orchestra.run.smithi070:> grep smithi141 /etc/hosts 2019-11-04T19:08:46.476 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.141 smithi141.front.sepia.ceph.com smithi141 2019-11-04T19:08:46.476 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:08:46.476 INFO:teuthology.orchestra.run.smithi070:> sudo hostname smithi070 2019-11-04T19:08:46.578 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:08:46.579 INFO:teuthology.orchestra.run.smithi070:> sudo sed -i -e 's/smithi141/smithi070/g' /etc/hosts 2019-11-04T19:08:46.638 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:08:46.638 INFO:teuthology.orchestra.run.smithi070:> sudo sed -i -e 's/172.21.15.141/172.21.15.70/g' /etc/hosts 2019-11-04T19:08:46.695 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:08:46.695 INFO:teuthology.orchestra.run.smithi070:> sudo sed -i -e 's/smithi141/smithi070/g' /etc/hostname 2019-11-04T19:08:46.751 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:08:46.751 INFO:teuthology.orchestra.run.smithi070:> sudo hostname smithi070 2019-11-04T19:08:46.804 INFO:teuthology.provision.fog.smithi070:Deploy complete! 2019-11-04T19:08:46.805 DEBUG:teuthology.parallel:result is None 2019-11-04T19:08:46.937 INFO:teuthology.lock.keys:Checking smithi070.front.sepia.ceph.com 2019-11-04T19:08:46.938 INFO:teuthology.lock.keys:New key found. Updating... 2019-11-04T19:08:46.959 INFO:teuthology.lock.ops:Updating [smithi070.front.sepia.ceph.com]: set os type and version on server 2019-11-04T19:08:46.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-11-04T19:08:46.960 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi070.front.sepia.ceph.com', 'timeout': 60} 2019-11-04T19:08:47.047 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:08:47.048 INFO:teuthology.orchestra.run.smithi070:> true 2019-11-04T19:08:47.319 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:08:47.319 INFO:teuthology.orchestra.run.smithi070:> uname -m 2019-11-04T19:08:47.367 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2019-11-04T19:08:47.368 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:08:47.368 INFO:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2019-11-04T19:08:47.413 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Linux" 2019-11-04T19:08:47.413 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="7 (Core)" 2019-11-04T19:08:47.413 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2019-11-04T19:08:47.413 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2019-11-04T19:08:47.413 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="7" 2019-11-04T19:08:47.413 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-11-04T19:08:47.414 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2019-11-04T19:08:47.414 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-11-04T19:08:47.414 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://www.centos.org/" 2019-11-04T19:08:47.414 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-11-04T19:08:47.414 INFO:teuthology.orchestra.run.smithi070.stdout: 2019-11-04T19:08:47.414 INFO:teuthology.orchestra.run.smithi070.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-11-04T19:08:47.415 INFO:teuthology.orchestra.run.smithi070.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-11-04T19:08:47.415 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-11-04T19:08:47.415 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-11-04T19:08:47.415 INFO:teuthology.orchestra.run.smithi070.stdout: 2019-11-04T19:08:47.416 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2019-11-04T19:08:47.429 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2019-11-04T19:08:47.449 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi070.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCm0EdEIZNIpKL55RPO+a7OOuOsO4mQJmQVpS7nq+PJiEhSIZrfweR8YQSmh9D+JgUCsRbCOg0qfYk36wFRIjoBEnFGaz2tflNKgzmzRJMaa8osZirJ3G3Aul5d/PaNemyhh6jWDBjixx+R2HBKwNUKZELoEoRiPgBq8AyqbMWvHWd3kQfx0cTFdJ6en4lOD9Z2qWkoP5w2OnNnbFMfYQd2VNpJzkiMcY3i5u+kqvGUXK7cOi3cdxl/dZ+5/HSmtLlc81R6MDdZf906G60DLXUJWveKunjAlRvzirQR9+3GEPrYIgV1S3dg+F3IDlYgBzcUzVfpnnfc4REUUrRRc9yJ 2019-11-04T19:08:47.449 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-11-04T19:08:47.495 INFO:teuthology.run_tasks:Running task internal.save_config... 2019-11-04T19:08:47.505 INFO:teuthology.task.internal:Saving configuration 2019-11-04T19:08:47.521 INFO:teuthology.run_tasks:Running task internal.check_lock... 2019-11-04T19:08:47.528 INFO:teuthology.task.internal.check_lock:Checking locks... 2019-11-04T19:08:47.542 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-11-04 19:03:57.181820', u'locked_by': u'scheduled_dang@sidious.arb.redhat.com', u'up': True, u'mac_address': None, u'name': u'smithi070.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHJ+K+p1S0cc3jiEgdvfSO4G3MYwbbsh+UlT+HR8hLpzM07hleAfNP7rubANbmTEwDg0+w0vOf5bGvHUKVH5h5M=', u'description': u'/home/teuthworker/archive/dang-2019-11-04_07:46:57-rgw-wip-dang-zipper-3-distro-basic-smithi/4472344'} 2019-11-04T19:08:47.542 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2019-11-04T19:08:47.549 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2019-11-04T19:08:47.549 INFO:teuthology.run_tasks:Running task console_log... 2019-11-04T19:08:47.647 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-11-04T19:08:47.649 INFO:teuthology.run_tasks:Running task internal.connect... 2019-11-04T19:08:47.661 INFO:teuthology.task.internal:Opening connections... 2019-11-04T19:08:47.661 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2019-11-04T19:08:47.663 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi070.front.sepia.ceph.com', 'timeout': 60} 2019-11-04T19:08:47.768 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2019-11-04T19:08:47.775 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:08:47.775 INFO:teuthology.orchestra.run.smithi070:> uname -m 2019-11-04T19:08:48.048 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2019-11-04T19:08:48.049 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:08:48.049 INFO:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2019-11-04T19:08:48.111 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Linux" 2019-11-04T19:08:48.112 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="7 (Core)" 2019-11-04T19:08:48.112 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2019-11-04T19:08:48.112 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2019-11-04T19:08:48.113 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="7" 2019-11-04T19:08:48.113 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-11-04T19:08:48.113 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2019-11-04T19:08:48.113 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-11-04T19:08:48.113 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://www.centos.org/" 2019-11-04T19:08:48.113 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-11-04T19:08:48.113 INFO:teuthology.orchestra.run.smithi070.stdout: 2019-11-04T19:08:48.114 INFO:teuthology.orchestra.run.smithi070.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-11-04T19:08:48.114 INFO:teuthology.orchestra.run.smithi070.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-11-04T19:08:48.114 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-11-04T19:08:48.114 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-11-04T19:08:48.115 INFO:teuthology.orchestra.run.smithi070.stdout: 2019-11-04T19:08:48.116 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2019-11-04T19:08:48.130 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2019-11-04T19:08:48.162 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2019-11-04T19:08:48.170 INFO:teuthology.task.internal:Checking for old test directory... 2019-11-04T19:08:48.171 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:08:48.171 INFO:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2019-11-04T19:08:48.216 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2019-11-04T19:08:48.243 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2019-11-04T19:08:48.244 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:08:48.244 INFO:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2019-11-04T19:08:48.297 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2019-11-04T19:08:48.343 INFO:teuthology.run_tasks:Running task kernel... 2019-11-04T19:08:48.367 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2019-11-04T19:08:48.367 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2019-11-04T19:08:48.368 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@smithi070.front.sepia.ceph.com 2019-11-04T19:08:48.368 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:08:48.368 INFO:teuthology.orchestra.run.smithi070:> sudo package-cleanup -y --oldkernels 2019-11-04T19:08:49.190 INFO:teuthology.orchestra.run.smithi070.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-11-04T19:09:20.612 INFO:teuthology.orchestra.run.smithi070.stdout:No old kernels to remove 2019-11-04T19:09:20.639 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:09:20.639 INFO:teuthology.orchestra.run.smithi070:> uname -r 2019-11-04T19:09:20.741 INFO:teuthology.orchestra.run.smithi070.stdout:3.10.0-957.27.2.el7.x86_64 2019-11-04T19:09:20.742 INFO:teuthology.task.kernel:Running kernel on smithi070: 3.10.0-957.27.2.el7.x86_64 2019-11-04T19:09:20.742 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:09:20.742 INFO:teuthology.orchestra.run.smithi070:> sudo yum install -y kernel 2019-11-04T19:09:20.986 INFO:teuthology.orchestra.run.smithi070.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-11-04T19:09:21.055 INFO:teuthology.orchestra.run.smithi070.stdout:Determining fastest mirrors 2019-11-04T19:09:21.317 INFO:teuthology.orchestra.run.smithi070.stdout: * base: mirror.cogentco.com 2019-11-04T19:09:21.317 INFO:teuthology.orchestra.run.smithi070.stdout: * epel: ftp.linux.ncsu.edu 2019-11-04T19:09:21.319 INFO:teuthology.orchestra.run.smithi070.stdout: * extras: mirror.fileplanet.com 2019-11-04T19:09:21.319 INFO:teuthology.orchestra.run.smithi070.stdout: * updates: mirror.cc.columbia.edu 2019-11-04T19:09:23.834 INFO:teuthology.orchestra.run.smithi070.stdout:1 packages excluded due to repository priority protections 2019-11-04T19:09:24.169 INFO:teuthology.orchestra.run.smithi070.stdout:Resolving Dependencies 2019-11-04T19:09:24.170 INFO:teuthology.orchestra.run.smithi070.stdout:--> Running transaction check 2019-11-04T19:09:24.170 INFO:teuthology.orchestra.run.smithi070.stdout:---> Package kernel.x86_64 0:3.10.0-1062.4.1.el7 will be installed 2019-11-04T19:09:24.395 INFO:teuthology.orchestra.run.smithi070.stdout:--> Processing Dependency: linux-firmware >= 20190429-72 for package: kernel-3.10.0-1062.4.1.el7.x86_64 2019-11-04T19:09:24.493 INFO:teuthology.orchestra.run.smithi070.stdout:--> Running transaction check 2019-11-04T19:09:24.493 INFO:teuthology.orchestra.run.smithi070.stdout:---> Package linux-firmware.noarch 0:20180911-69.git85c5d90.el7 will be updated 2019-11-04T19:09:24.519 INFO:teuthology.orchestra.run.smithi070.stdout:---> Package linux-firmware.noarch 0:20190429-72.gitddde598.el7 will be an update 2019-11-04T19:09:25.614 INFO:teuthology.orchestra.run.smithi070.stdout:--> Finished Dependency Resolution 2019-11-04T19:09:26.262 INFO:teuthology.orchestra.run.smithi070.stdout: 2019-11-04T19:09:26.262 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies Resolved 2019-11-04T19:09:26.265 INFO:teuthology.orchestra.run.smithi070.stdout: 2019-11-04T19:09:26.265 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2019-11-04T19:09:26.266 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2019-11-04T19:09:26.266 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2019-11-04T19:09:26.266 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2019-11-04T19:09:26.266 INFO:teuthology.orchestra.run.smithi070.stdout: kernel x86_64 3.10.0-1062.4.1.el7 updates 50 M 2019-11-04T19:09:26.266 INFO:teuthology.orchestra.run.smithi070.stdout:Updating for dependencies: 2019-11-04T19:09:26.266 INFO:teuthology.orchestra.run.smithi070.stdout: linux-firmware noarch 20190429-72.gitddde598.el7 base 73 M 2019-11-04T19:09:26.267 INFO:teuthology.orchestra.run.smithi070.stdout: 2019-11-04T19:09:26.267 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2019-11-04T19:09:26.267 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2019-11-04T19:09:26.267 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2019-11-04T19:09:26.267 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrade ( 1 Dependent package) 2019-11-04T19:09:26.267 INFO:teuthology.orchestra.run.smithi070.stdout: 2019-11-04T19:09:26.267 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 123 M 2019-11-04T19:09:26.267 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading packages: 2019-11-04T19:09:26.268 INFO:teuthology.orchestra.run.smithi070.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-11-04T19:09:27.886 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2019-11-04T19:09:27.886 INFO:teuthology.orchestra.run.smithi070.stdout:Total 76 MB/s | 123 MB 00:01 2019-11-04T19:09:27.958 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2019-11-04T19:09:28.149 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2019-11-04T19:09:29.038 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded 2019-11-04T19:09:29.038 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2019-11-04T19:09:39.609 INFO:teuthology.orchestra.run.smithi070.stdout: Updating : linux-firmware-20190429-72.gitddde598.el7.noarch 1/3 2019-11-04T19:09:44.130 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-3.10.0-1062.4.1.el7.x86_64 2/3 2019-11-04T19:10:12.334 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2019-11-04T19:10:12.493 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-3.10.0-1062.4.1.el7.x86_64 1/3 2019-11-04T19:10:12.618 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : linux-firmware-20190429-72.gitddde598.el7.noarch 2/3 2019-11-04T19:10:13.006 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2019-11-04T19:10:13.006 INFO:teuthology.orchestra.run.smithi070.stdout: 2019-11-04T19:10:13.007 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2019-11-04T19:10:13.007 INFO:teuthology.orchestra.run.smithi070.stdout: kernel.x86_64 0:3.10.0-1062.4.1.el7 2019-11-04T19:10:13.007 INFO:teuthology.orchestra.run.smithi070.stdout: 2019-11-04T19:10:13.007 INFO:teuthology.orchestra.run.smithi070.stdout:Dependency Updated: 2019-11-04T19:10:13.007 INFO:teuthology.orchestra.run.smithi070.stdout: linux-firmware.noarch 0:20190429-72.gitddde598.el7 2019-11-04T19:10:13.007 INFO:teuthology.orchestra.run.smithi070.stdout: 2019-11-04T19:10:13.007 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2019-11-04T19:10:13.144 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:10:13.145 INFO:teuthology.orchestra.run.smithi070:> rpm -q kernel --last | head -n 1 2019-11-04T19:10:13.534 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-3.10.0-1062.4.1.el7.x86_64 Mon 04 Nov 2019 07:09:39 PM UTC 2019-11-04T19:10:13.537 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.4.1.el7.x86_64 2019-11-04T19:10:13.537 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-957.27.2.el7.x86_64 Expected: 3.10.0-1062.4.1.el7.x86_64 2019-11-04T19:10:13.537 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2019-11-04T19:10:13.537 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-11-04T19:10:13.537 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2019-11-04T19:10:13.537 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi070.front.sepia.ceph.com, path=None, version=distro) 2019-11-04T19:10:13.538 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:10:13.538 INFO:teuthology.orchestra.run.smithi070:> rpm -q kernel --last | head -n 1 2019-11-04T19:10:13.825 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-3.10.0-1062.4.1.el7.x86_64 Mon 04 Nov 2019 07:09:39 PM UTC 2019-11-04T19:10:13.827 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.4.1.el7.x86_64 2019-11-04T19:10:13.827 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:10:13.827 INFO:teuthology.orchestra.run.smithi070:> sudo rpm -qi grub2 2019-11-04T19:10:13.949 INFO:teuthology.orchestra.run.smithi070.stdout:Name : grub2 2019-11-04T19:10:13.949 INFO:teuthology.orchestra.run.smithi070.stdout:Epoch : 1 2019-11-04T19:10:13.949 INFO:teuthology.orchestra.run.smithi070.stdout:Version : 2.02 2019-11-04T19:10:13.949 INFO:teuthology.orchestra.run.smithi070.stdout:Release : 0.76.el7.centos 2019-11-04T19:10:13.949 INFO:teuthology.orchestra.run.smithi070.stdout:Architecture: x86_64 2019-11-04T19:10:13.949 INFO:teuthology.orchestra.run.smithi070.stdout:Install Date: Wed 14 Aug 2019 05:05:05 PM UTC 2019-11-04T19:10:13.950 INFO:teuthology.orchestra.run.smithi070.stdout:Group : System Environment/Base 2019-11-04T19:10:13.950 INFO:teuthology.orchestra.run.smithi070.stdout:Size : 0 2019-11-04T19:10:13.950 INFO:teuthology.orchestra.run.smithi070.stdout:License : GPLv3+ 2019-11-04T19:10:13.950 INFO:teuthology.orchestra.run.smithi070.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:06 PM UTC, Key ID 24c6a8a7f4a80eb5 2019-11-04T19:10:13.950 INFO:teuthology.orchestra.run.smithi070.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2019-11-04T19:10:13.950 INFO:teuthology.orchestra.run.smithi070.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2019-11-04T19:10:13.951 INFO:teuthology.orchestra.run.smithi070.stdout:Build Host : kbuilder.bsys.centos.org 2019-11-04T19:10:13.951 INFO:teuthology.orchestra.run.smithi070.stdout:Relocations : (not relocatable) 2019-11-04T19:10:13.951 INFO:teuthology.orchestra.run.smithi070.stdout:Packager : CentOS BuildSystem 2019-11-04T19:10:13.951 INFO:teuthology.orchestra.run.smithi070.stdout:Vendor : CentOS 2019-11-04T19:10:13.951 INFO:teuthology.orchestra.run.smithi070.stdout:URL : http://www.gnu.org/software/grub/ 2019-11-04T19:10:13.951 INFO:teuthology.orchestra.run.smithi070.stdout:Summary : Bootloader with support for Linux, Multiboot and more 2019-11-04T19:10:13.952 INFO:teuthology.orchestra.run.smithi070.stdout:Description : 2019-11-04T19:10:13.952 INFO:teuthology.orchestra.run.smithi070.stdout: 2019-11-04T19:10:13.952 INFO:teuthology.orchestra.run.smithi070.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2019-11-04T19:10:13.952 INFO:teuthology.orchestra.run.smithi070.stdout:customizable bootloader with modular architecture. It supports a rich 2019-11-04T19:10:13.952 INFO:teuthology.orchestra.run.smithi070.stdout:variety of kernel formats, file systems, computer architectures and 2019-11-04T19:10:13.952 INFO:teuthology.orchestra.run.smithi070.stdout:hardware devices. 2019-11-04T19:10:13.953 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2019-11-04T19:10:13.954 INFO:teuthology.task.kernel:Updating grub on smithi070 to boot 3.10.0-1062.4.1.el7.x86_64 2019-11-04T19:10:13.954 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:10:13.954 INFO:teuthology.orchestra.run.smithi070:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2019-11-04T19:10:14.420 INFO:teuthology.orchestra.run.smithi070.stderr:Generating grub configuration file ... 2019-11-04T19:10:14.589 INFO:teuthology.orchestra.run.smithi070.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.4.1.el7.x86_64 2019-11-04T19:10:14.610 INFO:teuthology.orchestra.run.smithi070.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.4.1.el7.x86_64.img 2019-11-04T19:10:14.870 INFO:teuthology.orchestra.run.smithi070.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.27.2.el7.x86_64 2019-11-04T19:10:14.884 INFO:teuthology.orchestra.run.smithi070.stderr:Found initrd image: /boot/initramfs-3.10.0-957.27.2.el7.x86_64.img 2019-11-04T19:10:14.915 INFO:teuthology.orchestra.run.smithi070.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2019-11-04T19:10:14.935 INFO:teuthology.orchestra.run.smithi070.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2019-11-04T19:10:14.962 INFO:teuthology.orchestra.run.smithi070.stderr:Found linux image: /boot/vmlinuz-0-rescue-48f603b7a56f4bb39f72becf12f3b725 2019-11-04T19:10:14.971 INFO:teuthology.orchestra.run.smithi070.stderr:Found initrd image: /boot/initramfs-0-rescue-48f603b7a56f4bb39f72becf12f3b725.img 2019-11-04T19:10:16.026 INFO:teuthology.orchestra.run.smithi070.stderr:done 2019-11-04T19:10:16.026 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:10:16.027 INFO:teuthology.orchestra.run.smithi070:> mktemp 2019-11-04T19:10:16.080 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.8JyBtjIcfm 2019-11-04T19:10:16.081 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:10:16.081 INFO:teuthology.orchestra.run.smithi070:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8JyBtjIcfm 2019-11-04T19:10:16.144 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:10:16.144 INFO:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.8JyBtjIcfm 2019-11-04T19:10:16.258 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.8JyBtjIcfm is 6KB 2019-11-04T19:10:16.309 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:10:16.309 INFO:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.8JyBtjIcfm 2019-11-04T19:10:16.359 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:10:16.359 INFO:teuthology.orchestra.run.smithi070:> sudo grub2-set-default 0 2019-11-04T19:10:16.614 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:10:16.614 INFO:teuthology.orchestra.run.smithi070:> sudo shutdown -r now 2019-11-04T19:10:16.619 INFO:teuthology.misc:Re-opening connections... 2019-11-04T19:10:16.620 INFO:teuthology.misc:trying to connect to ubuntu@smithi070.front.sepia.ceph.com 2019-11-04T19:10:16.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-11-04T19:10:16.622 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi070.front.sepia.ceph.com', 'timeout': 60} 2019-11-04T19:10:16.733 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:10:16.733 INFO:teuthology.orchestra.run.smithi070:> true 2019-11-04T19:10:17.049 DEBUG:teuthology.misc:waited 0.429113149643 2019-11-04T19:10:18.050 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2019-11-04T19:10:18.050 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "3.10.0-1062.4.1.el7.x86_64"... 2019-11-04T19:10:18.050 INFO:teuthology.orchestra.run.smithi070:Running: 2019-11-04T19:10:18.050 INFO:teuthology.orchestra.run.smithi070:> uname -r 2019-11-05T07:04:15.757 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2019-11-05T07:04:15.770 DEBUG:teuthology.task.console_log:Killing console logger for smithi070 2019-11-05T07:04:15.771 DEBUG:teuthology.task.console_log:Killing console logger for smithi070 2019-11-05T07:04:15.771 DEBUG:teuthology.exit:Finished running handlers