2021-02-10T22:20:47.438 INFO:root:teuthology version: 1.1.0-14fad317 2021-02-10T22:20:47.440 DEBUG:teuthology.run:Teuthology command: teuthology --name nojha-2021-02-10_22:11:28-rados:cephadm-master-distro-basic-smithi --description rados:cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} --verbose --archive /home/teuthworker/archive/nojha-2021-02-10_22:11:28-rados:cephadm-master-distro-basic-smithi/5873913 --owner scheduled_nojha@teuthology -- /home/teuthworker/archive/nojha-2021-02-10_22:11:28-rados:cephadm-master-distro-basic-smithi/5873913/orig.config.yaml 2021-02-10T22:20:47.482 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-02-10T22:20:47.543 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/nojha-2021-02-10_22:11:28-rados:cephadm-master-distro-basic-smithi/5873913 branch: master description: rados:cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} email: null first_in_suite: false job_id: '5873913' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: nojha-2021-02-10_22:11:28-rados:cephadm-master-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: '8.0' overrides: admin_socket: branch: master ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd shutdown pgref assert: true log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: b4d9cc45d6ff1ea5382954dece424128b478d6f7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: b4d9cc45d6ff1ea5382954dece424128b478d6f7 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-cephadm-compress sha1: 3ced450cdd7a1f4083a19dd29b2d27e03883ab52 owner: scheduled_nojha@teuthology priority: 19 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.realm.zone.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a sha1: b4d9cc45d6ff1ea5382954dece424128b478d6f7 sleep_before_teardown: 0 suite: rados:cephadm suite_branch: wip-cephadm-compress suite_path: /home/teuthworker/src/github.com_neha-ojha_ceph_3ced450cdd7a1f4083a19dd29b2d27e03883ab52/qa suite_relpath: qa suite_repo: https://github.com/neha-ojha/ceph.git suite_sha1: 3ced450cdd7a1f4083a19dd29b2d27e03883ab52 targets: smithi087.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5yhxLsWFGDZQoKxIKpOxHAkRkFnkdyrmjCJQusAEsAKyqyMDD9nIbj4wJxoFytQmrczDAYvUfcISae6pa9wdiFv8M7t24v/jXroXlbBgI6v5Lqbo7qT3jVKTSJbv8Kya2NT5GKV+ANgDvN5V7TG3JfcC7jerSuQAn5g2zJYLaezT6/SdYaVI767SpWHpNM1g/xjKtByZRa1dR7pUQP4pDtaPM7dTi4Qgma2/UBTunD4bX9XJVm0rWy9lU2Pjo9ZIfwr+OtiXFC4p6rT4l8ay8raqJ7UfebokEnx5CisKJgkMWqzvsOe32c/eB2PYs6WsSj0i+MtbLexeUICef7NJsySCZ4T8bGkVBp+tTBolhT+U25Cq0MuUwZ8TjeghTCUPtZtKHiopP2g/gzaZ/ZUtNS5nInQuTaDtXCsKNo6F88skpS4SD8sXUSjhMMx4+SLoIObGmbYkCNs85UyXSGpeHjojN8RNFzwcJrnH9K07ARq82EgWPtHlRR9LMxgpYeSE= smithi146.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCh7e1rq1m7sZOxFSDkYvRF37hNRz2IEt0+npFtlmXdq+3JKw7C+lVQ5rq6R5iFkfZeKlFCRMe2pndLYfLn/TyKQ9v/3+1lP3UMThGzQHZW0o+R/3WBA95M/EAnAhjsyi6KSXyPgfJVxWrK8VpGv0Anh3hFxp3qh+P4AhfsTVphqzKvPDOPcUgbwn6R1H5WabkrMHLt8uceu1rntK+n0dE0dTiXRswlGLC/N89dP3xh7B3vVLMnbQE+NQyRhXjUYjj8q7QqmtoUIdietX2U/s8725fsacdM/gMxDhaA91zfjiuJ2vkBcEEm1eCDX22QXFsbvfV7owMQYobfnf7+iqLh8a/B/MrdB/lNuRUc2fHzKUJl/BuUwDbynfmqDgZkhej022nuCSwyqVtzB+IbYdVdsZ2MD5EckQ8GPj9tIAcofkiRJGEuAx7+bMZxwBSp+wt1NM/6I41k1eVEcreLMlvDDl1egB0v1R9t6kzdCc1Q/XFke+rAnZ94Ai6k8jVvmqk= tasks: - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml teuthology_branch: master teuthology_sha1: 14fad31725ce3e5fc27890f2a8ef1f21c838d296 timestamp: 2021-02-10_22:11:28 tube: smithi user: nojha verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.29888 2021-02-10T22:20:47.543 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_neha-ojha_ceph_3ced450cdd7a1f4083a19dd29b2d27e03883ab52/qa; will attempt to use it 2021-02-10T22:20:47.543 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_neha-ojha_ceph_3ced450cdd7a1f4083a19dd29b2d27e03883ab52/qa/tasks 2021-02-10T22:20:47.544 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-02-10T22:20:47.545 INFO:teuthology.task.internal:Checking packages... 2021-02-10T22:20:47.569 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'b4d9cc45d6ff1ea5382954dece424128b478d6f7' 2021-02-10T22:20:47.569 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-02-10T22:20:47.569 INFO:teuthology.packaging:ref: None 2021-02-10T22:20:47.569 INFO:teuthology.packaging:tag: None 2021-02-10T22:20:47.569 INFO:teuthology.packaging:branch: master 2021-02-10T22:20:47.569 INFO:teuthology.packaging:sha1: b4d9cc45d6ff1ea5382954dece424128b478d6f7 2021-02-10T22:20:47.570 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=master 2021-02-10T22:20:48.462 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-703.gb4d9cc45 2021-02-10T22:20:48.462 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-02-10T22:20:48.468 INFO:teuthology.task.internal:no buildpackages task found 2021-02-10T22:20:48.468 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-02-10T22:20:48.953 INFO:teuthology.task.internal:Saving configuration 2021-02-10T22:20:48.964 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-02-10T22:20:49.011 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-10T22:20:49.029 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-10 22:15:04.727640', 'description': '/home/teuthworker/archive/nojha-2021-02-10_22:11:28-rados:cephadm-master-distro-basic-smithi/5873913', 'locked_by': 'scheduled_nojha@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFAsKNCfH/7fIGjywLSQ2SSmALY7ApogIRW8SKpRVGjA0+gBhP7F6iwf93YoUb90fuhZFOMHP5GA/+Cf+TlZAC8=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi087.front.sepia.ceph.com'} 2021-02-10T22:20:49.046 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-10 22:15:04.727618', 'description': '/home/teuthworker/archive/nojha-2021-02-10_22:11:28-rados:cephadm-master-distro-basic-smithi/5873913', 'locked_by': 'scheduled_nojha@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBII3CvVKKQwABTMbh8d3djcKCfGFU1pZVvz+QSOtvCr9sTj6RyF3f311jQlfgFr8kIGdv432UzUiFlD4pU3qZhE=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi146.front.sepia.ceph.com'} 2021-02-10T22:20:49.046 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-02-10T22:20:49.065 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.realm.zone.a', 'node-exporter.a', 'alertmanager.a'] 2021-02-10T22:20:49.065 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2021-02-10T22:20:49.065 INFO:teuthology.run_tasks:Running task console_log... 2021-02-10T22:20:49.305 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdf09608ae8>, signals=[15]) 2021-02-10T22:20:49.306 INFO:teuthology.run_tasks:Running task internal.connect... 2021-02-10T22:20:49.402 INFO:teuthology.task.internal:Opening connections... 2021-02-10T22:20:49.402 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2021-02-10T22:20:49.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T22:20:49.482 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2021-02-10T22:20:49.484 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T22:20:49.560 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-02-10T22:20:49.946 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2021-02-10T22:20:49.966 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2021-02-10T22:20:49.967 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2021-02-10T22:20:50.026 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Linux" 2021-02-10T22:20:50.026 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="8 (Core)" 2021-02-10T22:20:50.027 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2021-02-10T22:20:50.027 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2021-02-10T22:20:50.027 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="8" 2021-02-10T22:20:50.027 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el8" 2021-02-10T22:20:50.027 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-02-10T22:20:50.027 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2021-02-10T22:20:50.027 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-02-10T22:20:50.027 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://www.centos.org/" 2021-02-10T22:20:50.028 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-10T22:20:50.028 INFO:teuthology.orchestra.run.smithi087.stdout: 2021-02-10T22:20:50.028 INFO:teuthology.orchestra.run.smithi087.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-02-10T22:20:50.028 INFO:teuthology.orchestra.run.smithi087.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-02-10T22:20:50.028 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-10T22:20:50.028 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-02-10T22:20:50.028 INFO:teuthology.orchestra.run.smithi087.stdout: 2021-02-10T22:20:50.030 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2021-02-10T22:20:50.060 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2021-02-10T22:20:50.078 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2021-02-10T22:20:50.079 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2021-02-10T22:20:50.134 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Linux" 2021-02-10T22:20:50.134 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="8 (Core)" 2021-02-10T22:20:50.135 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2021-02-10T22:20:50.135 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2021-02-10T22:20:50.136 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="8" 2021-02-10T22:20:50.136 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el8" 2021-02-10T22:20:50.136 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-02-10T22:20:50.136 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2021-02-10T22:20:50.136 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-02-10T22:20:50.136 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.centos.org/" 2021-02-10T22:20:50.137 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-10T22:20:50.137 INFO:teuthology.orchestra.run.smithi146.stdout: 2021-02-10T22:20:50.137 INFO:teuthology.orchestra.run.smithi146.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-02-10T22:20:50.137 INFO:teuthology.orchestra.run.smithi146.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-02-10T22:20:50.137 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-10T22:20:50.137 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-02-10T22:20:50.138 INFO:teuthology.orchestra.run.smithi146.stdout: 2021-02-10T22:20:50.138 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2021-02-10T22:20:50.157 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-02-10T22:20:50.340 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-02-10T22:20:50.420 INFO:teuthology.task.internal:Checking for old test directory... 2021-02-10T22:20:50.428 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2021-02-10T22:20:50.434 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2021-02-10T22:20:50.457 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-02-10T22:20:50.582 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-02-10T22:20:50.583 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2021-02-10T22:20:50.589 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2021-02-10T22:20:50.604 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-02-10T22:20:50.666 INFO:teuthology.run_tasks:Running task kernel... 2021-02-10T22:20:50.688 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'ceph.rgw.realm.zone.a': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.a': {'kdb': True, 'sha1': 'distro'}, 'alertmanager.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'prometheus.a': {'kdb': True, 'sha1': 'distro'}, 'grafana.a': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.b': {'kdb': True, 'sha1': 'distro'}, 'ceph.iscsi.iscsi.a': {'kdb': True, 'sha1': 'distro'}} 2021-02-10T22:20:50.688 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-02-10T22:20:50.688 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2021-02-10T22:20:50.714 INFO:teuthology.orchestra.run.smithi087.stdout:4.18.0-147.8.1.el8_1.x86_64 2021-02-10T22:20:50.714 INFO:teuthology.task.kernel:Running kernel on smithi087: 4.18.0-147.8.1.el8_1.x86_64 2021-02-10T22:20:50.715 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum install -y kernel 2021-02-10T22:20:53.346 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-8 - AppStream 31 MB/s | 6.3 MB 00:00 2021-02-10T22:20:54.434 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-8 - Base 23 MB/s | 2.3 MB 00:00 2021-02-10T22:20:54.881 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-8 - Extras 137 kB/s | 8.6 kB 00:00 2021-02-10T22:20:55.007 INFO:teuthology.orchestra.run.smithi087.stdout:Copr repo for ceph-el8 owned by ktdreyer 41 kB/s | 2.9 kB 00:00 2021-02-10T22:20:55.417 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 25 MB/s | 8.8 MB 00:00 2021-02-10T22:20:57.274 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2021-02-10T22:20:58.327 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-4.18.0-147.8.1.el8_1.x86_64 is already installed. 2021-02-10T22:20:58.328 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2021-02-10T22:20:58.489 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2021-02-10T22:20:58.490 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2021-02-10T22:20:58.490 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2021-02-10T22:20:58.490 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2021-02-10T22:20:58.490 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2021-02-10T22:20:58.490 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 4.18.0-240.10.1.el8_3 CentOS-Base 4.3 M 2021-02-10T22:20:58.490 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrading: 2021-02-10T22:20:58.490 INFO:teuthology.orchestra.run.smithi087.stdout: kexec-tools x86_64 2.0.20-34.el8_3.1 CentOS-Base 496 k 2021-02-10T22:20:58.490 INFO:teuthology.orchestra.run.smithi087.stdout: linux-firmware noarch 20200619-101.git3890db36.el8_3 CentOS-Base 101 M 2021-02-10T22:20:58.490 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2021-02-10T22:20:58.491 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core x86_64 4.18.0-240.10.1.el8_3 CentOS-Base 30 M 2021-02-10T22:20:58.491 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules x86_64 4.18.0-240.10.1.el8_3 CentOS-Base 26 M 2021-02-10T22:20:58.491 INFO:teuthology.orchestra.run.smithi087.stdout: 2021-02-10T22:20:58.491 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2021-02-10T22:20:58.491 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2021-02-10T22:20:58.491 INFO:teuthology.orchestra.run.smithi087.stdout:Install 3 Packages 2021-02-10T22:20:58.491 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrade 2 Packages 2021-02-10T22:20:58.491 INFO:teuthology.orchestra.run.smithi087.stdout: 2021-02-10T22:20:58.492 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 162 M 2021-02-10T22:20:58.492 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2021-02-10T22:20:58.738 INFO:teuthology.orchestra.run.smithi087.stdout:(1/5): kernel-4.18.0-240.10.1.el8_3.x86_64.rpm 18 MB/s | 4.3 MB 00:00 2021-02-10T22:20:58.758 INFO:teuthology.orchestra.run.smithi087.stdout:(2/5): kexec-tools-2.0.20-34.el8_3.1.x86_64.rpm 23 MB/s | 496 kB 00:00 2021-02-10T22:20:59.217 INFO:teuthology.orchestra.run.smithi087.stdout:(3/5): kernel-core-4.18.0-240.10.1.el8_3.x86_64 42 MB/s | 30 MB 00:00 2021-02-10T22:20:59.686 INFO:teuthology.orchestra.run.smithi087.stdout:(4/5): kernel-modules-4.18.0-240.10.1.el8_3.x86 21 MB/s | 26 MB 00:01 2021-02-10T22:21:01.004 INFO:teuthology.orchestra.run.smithi087.stdout:(5/5): linux-firmware-20200619-101.git3890db36. 45 MB/s | 101 MB 00:02 2021-02-10T22:21:01.007 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2021-02-10T22:21:01.007 INFO:teuthology.orchestra.run.smithi087.stdout:Total 64 MB/s | 162 MB 00:02 2021-02-10T22:21:01.100 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2021-02-10T22:21:01.308 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2021-02-10T22:21:01.308 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2021-02-10T22:21:02.942 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2021-02-10T22:21:02.943 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2021-02-10T22:21:04.335 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2021-02-10T22:21:12.946 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: linux-firmware-20200619-101.git3890db36.el8_3.noarch 1/1 2021-02-10T22:21:14.644 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : linux-firmware-20200619-101.git3890db36.el8_3.noarch 1/7 2021-02-10T22:21:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-core-4.18.0-240.10.1.el8_3.x86_64 2/7 2021-02-10T22:21:17.810 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-4.18.0-240.10.1.el8_3.x86_64 2/7 2021-02-10T22:21:18.159 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-4.18.0-240.10.1.el8_3.x86_64 3/7 2021-02-10T22:21:21.034 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-4.18.0-240.10.1.el8_3.x86_64 3/7 2021-02-10T22:21:21.334 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-4.18.0-240.10.1.el8_3.x86_64 4/7 2021-02-10T22:21:21.575 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : kexec-tools-2.0.20-34.el8_3.1.x86_64 5/7 2021-02-10T22:21:21.629 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kexec-tools-2.0.20-34.el8_3.1.x86_64 5/7 2021-02-10T22:21:21.865 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : linux-firmware-20190516-94.git711d3297.el8.noarch 6/7 2021-02-10T22:21:21.882 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kexec-tools-2.0.19-12.el8_1.2.x86_64 7/7 2021-02-10T22:21:21.882 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : kexec-tools-2.0.19-12.el8_1.2.x86_64 7/7 2021-02-10T22:21:30.147 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kexec-tools-2.0.19-12.el8_1.2.x86_64 7/7 2021-02-10T22:21:49.612 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-4.18.0-240.10.1.el8_3.x86_64 7/7 2021-02-10T22:21:50.060 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kexec-tools-2.0.19-12.el8_1.2.x86_64 7/7 2021-02-10T22:21:50.060 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-4.18.0-240.10.1.el8_3.x86_64 1/7 2021-02-10T22:21:50.060 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-core-4.18.0-240.10.1.el8_3.x86_64 2/7 2021-02-10T22:21:50.060 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-4.18.0-240.10.1.el8_3.x86_64 3/7 2021-02-10T22:21:50.060 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kexec-tools-2.0.20-34.el8_3.1.x86_64 4/7 2021-02-10T22:21:50.061 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kexec-tools-2.0.19-12.el8_1.2.x86_64 5/7 2021-02-10T22:21:50.061 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : linux-firmware-20200619-101.git3890db36.el8_3.noarch 6/7 2021-02-10T22:21:50.272 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-10T22:21:50.641 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : linux-firmware-20190516-94.git711d3297.el8.noarch 7/7 2021-02-10T22:21:50.642 INFO:teuthology.orchestra.run.smithi087.stdout: 2021-02-10T22:21:50.642 INFO:teuthology.orchestra.run.smithi087.stdout:Upgraded: 2021-02-10T22:21:50.642 INFO:teuthology.orchestra.run.smithi087.stdout: kexec-tools-2.0.20-34.el8_3.1.x86_64 2021-02-10T22:21:50.642 INFO:teuthology.orchestra.run.smithi087.stdout: linux-firmware-20200619-101.git3890db36.el8_3.noarch 2021-02-10T22:21:50.642 INFO:teuthology.orchestra.run.smithi087.stdout: 2021-02-10T22:21:50.642 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2021-02-10T22:21:50.642 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-4.18.0-240.10.1.el8_3.x86_64 2021-02-10T22:21:50.642 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core-4.18.0-240.10.1.el8_3.x86_64 2021-02-10T22:21:50.642 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-4.18.0-240.10.1.el8_3.x86_64 2021-02-10T22:21:50.642 INFO:teuthology.orchestra.run.smithi087.stdout: 2021-02-10T22:21:50.642 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2021-02-10T22:21:50.741 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel --last | head -n 1 2021-02-10T22:21:50.824 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-4.18.0-240.10.1.el8_3.x86_64 Wed 10 Feb 2021 10:21:21 PM UTC 2021-02-10T22:21:50.826 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.10.1.el8_3.x86_64 2021-02-10T22:21:50.826 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-147.8.1.el8_1.x86_64 Expected: 4.18.0-240.10.1.el8_3.x86_64 2021-02-10T22:21:50.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-10T22:21:50.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T22:21:50.828 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2021-02-10T22:21:50.829 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/task/kernel.py", line 1263, in task version = need_to_install_distro(role_remote) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/task/kernel.py", line 748, in need_to_install_distro current = get_version_of_running_kernel(remote) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/task/kernel.py", line 724, in get_version_of_running_kernel uname_r = remote.sh("uname -r").strip() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/remote.py", line 259, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi146 was lost: 'uname -r' 2021-02-10T22:21:51.109 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d2b6e8c911ed4cd9a85ad6039103bc20 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/task/kernel.py", line 1263, in task version = need_to_install_distro(role_remote) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/task/kernel.py", line 748, in need_to_install_distro current = get_version_of_running_kernel(remote) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/task/kernel.py", line 724, in get_version_of_running_kernel uname_r = remote.sh("uname -r").strip() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/remote.py", line 259, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi146 was lost: 'uname -r' 2021-02-10T22:21:51.111 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-02-10T22:21:51.600 INFO:teuthology.nuke:Checking targets against current locks 2021-02-10T22:21:51.730 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 412, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 269, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 266, in nuke for unnuked in p: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 280, in nuke_one keep_logs = ctx.keep_logs File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/config.py", line 255, in __getattr__ raise AttributeError(name) AttributeError: keep_logs