2021-10-17T15:28:14.832 INFO:root:teuthology version: 1.1.0-b35344e8 2021-10-17T15:28:14.834 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2021-10-17_15:11:50-rados-wip-yuri2-testing-2021-10-15-1332-pacific-distro-basic-smithi/6447679 --owner scheduled_yuriw@teuthology --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} --name yuriw-2021-10-17_15:11:50-rados-wip-yuri2-testing-2021-10-15-1332-pacific-distro-basic-smithi --verbose -- /home/teuthworker/archive/yuriw-2021-10-17_15:11:50-rados-wip-yuri2-testing-2021-10-15-1332-pacific-distro-basic-smithi/6447679/orig.config.yaml 2021-10-17T15:28:14.879 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-10-17T15:28:14.942 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2021-10-17_15:11:50-rados-wip-yuri2-testing-2021-10-15-1332-pacific-distro-basic-smithi/6447679 branch: wip-yuri2-testing-2021-10-15-1332-pacific description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '6447679' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2021-10-17_15:11:50-rados-wip-yuri2-testing-2021-10-15-1332-pacific-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: '8.2' overrides: admin_socket: branch: wip-yuri2-testing-2021-10-15-1332-pacific ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 15f9519d2bb8c31d5d6afe04bfbd467cba5da03f valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: debuginfo: true sha1: 15f9519d2bb8c31d5d6afe04bfbd467cba5da03f selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri2-testing-2021-10-15-1332-pacific sha1: 15f9519d2bb8c31d5d6afe04bfbd467cba5da03f owner: scheduled_yuriw@teuthology priority: 79 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b sha1: 15f9519d2bb8c31d5d6afe04bfbd467cba5da03f sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri2-testing-2021-10-15-1332-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_15f9519d2bb8c31d5d6afe04bfbd467cba5da03f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 15f9519d2bb8c31d5d6afe04bfbd467cba5da03f targets: smithi140.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwr6mkQ4d4VqW/NpPn14Suc5a3hzZIOl0FHEoJBieP4YES2jeK7AYXrAehRUplXuzIyX3t7sW1f14nRKxdaQS6EpLhTlp5lWSjdYEewSAe3PBkXgLrmA0jW7BEGOI6ApCFc9P0Hj3L8LzHMUau0AQHBLpicoEZiDliRy3uOkzkU01/GNvir31KTDD4fhoIqJEieqUu/XFSFgXBkJeYtRZ70SyNlFnhYybizhXreiXHMhEyKaOCKx76xkriRspbcm3n8wNDPX7n8H3tzgLw70Hz+zpweSJ6B66c9t+DAi5yq98s4dmU8VmjmfTR3mF8OrNeLCxG0lJ7mQSxbNTcli3ywfO+Y3ZlVLdHd1PBVu9SHAF0UiTXsKcuE6s54I6vOZMhulF/fOlWNliHiPQgrz+PyAVt/Pvw8tIttYPaldYSytFKMEhJtrSi9RcHixW8LEdxb9lZNWze0jMPpp0aefJj9sVgjE0ho13NnCjpV45cR/ZVvX4D+XFaDCG4dxG8xok= smithi194.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIpvVw9U1SlCventvo+UbgHBT1x1d4QCGyRrrqcySWSq2GUn4PoC9HlezADIqYIigfldMDC/5Q4lrel3zLPyswx0j2sMP4+yF57IAtp9+Nhe7kuBp25LdC0M6Motd1Ygjx3wUtgWoN6MwOKftwBWVMV6qb5NAUD5498xdBl21mV+0JQ+GSsn2D3mANSljclpqCY8QXTZpDZ9bLn0CK+BiEV8x0kBwvQjzZ7tsufcUP20DkCULCt90T4lylm+7EhXqJuGK0r53BJjqRCSnovzuFXO1QxwbleEZTW5CsQLXnVmPIHXFYhkt3EATQvB+KHtw6h9PwcGkvMsy6P6sUqQkt3B0G2gCqpYvtVz3rthi6uj/DZuQVychfOBUoOf9bcbxXkWamHwsaIVD4ywWEOQucprIBK/d4iduMzqNGgsTtvHbnmk5OmrHp5CdX0XOCE2DlDhVOi0e+bCHEee5co1pfPpQHl16CZVOdQdDgSpm55MsxV2DjPD//STH+Jc0mNrU= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology_branch: master teuthology_sha1: b35344e81ac507b6cad7c1cae575ce08b9c766f2 timestamp: 2021-10-17_15:11:50 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.10511 2021-10-17T15:28:14.943 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_15f9519d2bb8c31d5d6afe04bfbd467cba5da03f/qa; will attempt to use it 2021-10-17T15:28:14.943 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_15f9519d2bb8c31d5d6afe04bfbd467cba5da03f/qa/tasks 2021-10-17T15:28:14.944 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-10-17T15:28:14.946 INFO:teuthology.task.internal:Checking packages... 2021-10-17T15:28:14.969 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '15f9519d2bb8c31d5d6afe04bfbd467cba5da03f' 2021-10-17T15:28:14.969 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-10-17T15:28:14.970 INFO:teuthology.packaging:ref: None 2021-10-17T15:28:14.970 INFO:teuthology.packaging:tag: None 2021-10-17T15:28:14.970 INFO:teuthology.packaging:branch: wip-yuri2-testing-2021-10-15-1332-pacific 2021-10-17T15:28:14.971 INFO:teuthology.packaging:sha1: 15f9519d2bb8c31d5d6afe04bfbd467cba5da03f 2021-10-17T15:28:14.971 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2021-10-15-1332-pacific 2021-10-17T15:28:15.196 INFO:teuthology.task.internal:Found packages for ceph version 16.2.6-252.g15f9519d 2021-10-17T15:28:15.197 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-10-17T15:28:15.209 INFO:teuthology.task.internal:no buildpackages task found 2021-10-17T15:28:15.210 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-10-17T15:28:15.238 INFO:teuthology.task.internal:Saving configuration 2021-10-17T15:28:15.257 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-10-17T15:28:15.267 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-10-17T15:28:15.288 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi140.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2021-10-17_15:11:50-rados-wip-yuri2-testing-2021-10-15-1332-pacific-distro-basic-smithi/6447679', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-10-17 15:17:43.027872', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPLm3gg3qzfELQQHYkeOGgEbe357LVY3/Zl5rb9CEX45r9xTt0toMiCB0J7RgNgipHB0LoM843MbOk4J2aQYO/o='} 2021-10-17T15:28:15.308 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2021-10-17_15:11:50-rados-wip-yuri2-testing-2021-10-15-1332-pacific-distro-basic-smithi/6447679', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-10-17 15:17:43.027891', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEOu8XSjitpjEWJFsRtW7CYESnAL4tFLT/jDadc01ZHNzqqUw41Uy4iEemOGUGEUl76rjZQIdelWDBxN4qCVSzE='} 2021-10-17T15:28:15.308 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-10-17T15:28:15.316 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-10-17T15:28:15.316 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-10-17T15:28:15.316 INFO:teuthology.run_tasks:Running task console_log... 2021-10-17T15:28:15.398 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7e9ecd1d08>, signals=[15]) 2021-10-17T15:28:15.400 INFO:teuthology.run_tasks:Running task internal.connect... 2021-10-17T15:28:15.411 INFO:teuthology.task.internal:Opening connections... 2021-10-17T15:28:15.411 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi140.front.sepia.ceph.com 2021-10-17T15:28:15.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-10-17T15:28:15.490 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2021-10-17T15:28:15.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-10-17T15:28:15.608 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-10-17T15:28:15.619 DEBUG:teuthology.orchestra.run.smithi140:> uname -m 2021-10-17T15:28:15.640 INFO:teuthology.orchestra.run.smithi140.stdout:x86_64 2021-10-17T15:28:15.640 DEBUG:teuthology.orchestra.run.smithi140:> cat /etc/os-release 2021-10-17T15:28:15.700 INFO:teuthology.orchestra.run.smithi140.stdout:NAME="CentOS Linux" 2021-10-17T15:28:15.701 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION="8 (Core)" 2021-10-17T15:28:15.701 INFO:teuthology.orchestra.run.smithi140.stdout:ID="centos" 2021-10-17T15:28:15.701 INFO:teuthology.orchestra.run.smithi140.stdout:ID_LIKE="rhel fedora" 2021-10-17T15:28:15.701 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION_ID="8" 2021-10-17T15:28:15.702 INFO:teuthology.orchestra.run.smithi140.stdout:PLATFORM_ID="platform:el8" 2021-10-17T15:28:15.702 INFO:teuthology.orchestra.run.smithi140.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-10-17T15:28:15.702 INFO:teuthology.orchestra.run.smithi140.stdout:ANSI_COLOR="0;31" 2021-10-17T15:28:15.702 INFO:teuthology.orchestra.run.smithi140.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-10-17T15:28:15.702 INFO:teuthology.orchestra.run.smithi140.stdout:HOME_URL="https://www.centos.org/" 2021-10-17T15:28:15.703 INFO:teuthology.orchestra.run.smithi140.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-10-17T15:28:15.703 INFO:teuthology.orchestra.run.smithi140.stdout: 2021-10-17T15:28:15.703 INFO:teuthology.orchestra.run.smithi140.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-10-17T15:28:15.703 INFO:teuthology.orchestra.run.smithi140.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-10-17T15:28:15.704 INFO:teuthology.orchestra.run.smithi140.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-10-17T15:28:15.704 INFO:teuthology.orchestra.run.smithi140.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-10-17T15:28:15.704 INFO:teuthology.orchestra.run.smithi140.stdout: 2021-10-17T15:28:15.705 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server 2021-10-17T15:28:15.726 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2021-10-17T15:28:15.743 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2021-10-17T15:28:15.743 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2021-10-17T15:28:15.797 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Linux" 2021-10-17T15:28:15.798 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="8 (Core)" 2021-10-17T15:28:15.798 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2021-10-17T15:28:15.798 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2021-10-17T15:28:15.798 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="8" 2021-10-17T15:28:15.799 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el8" 2021-10-17T15:28:15.799 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-10-17T15:28:15.799 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2021-10-17T15:28:15.799 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-10-17T15:28:15.799 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://www.centos.org/" 2021-10-17T15:28:15.800 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-10-17T15:28:15.800 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-10-17T15:28:15.800 INFO:teuthology.orchestra.run.smithi194.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-10-17T15:28:15.800 INFO:teuthology.orchestra.run.smithi194.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-10-17T15:28:15.801 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-10-17T15:28:15.801 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-10-17T15:28:15.801 INFO:teuthology.orchestra.run.smithi194.stdout: 2021-10-17T15:28:15.802 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2021-10-17T15:28:15.820 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-10-17T15:28:15.834 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-10-17T15:28:15.867 INFO:teuthology.task.internal:Checking for old test directory... 2021-10-17T15:28:15.868 DEBUG:teuthology.orchestra.run.smithi140:> test '!' -e /home/ubuntu/cephtest 2021-10-17T15:28:15.870 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2021-10-17T15:28:15.884 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-10-17T15:28:15.895 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-10-17T15:28:15.895 DEBUG:teuthology.orchestra.run.smithi140:> test -z $(ls -A /var/lib/ceph) 2021-10-17T15:28:15.931 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2021-10-17T15:28:15.952 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-10-17T15:28:16.024 INFO:teuthology.run_tasks:Running task kernel... 2021-10-17T15:28:16.043 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2021-10-17T15:28:16.044 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-10-17T15:28:16.044 DEBUG:teuthology.orchestra.run.smithi140:> uname -r 2021-10-17T15:28:16.061 INFO:teuthology.orchestra.run.smithi140.stdout:4.18.0-305.12.1.el8_4.x86_64 2021-10-17T15:28:16.062 INFO:teuthology.task.kernel:Running kernel on smithi140: 4.18.0-305.12.1.el8_4.x86_64 2021-10-17T15:28:16.063 DEBUG:teuthology.orchestra.run.smithi140:> sudo yum install -y kernel 2021-10-17T15:28:18.582 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS-8 - AppStream 42 MB/s | 9.3 MB 00:00 2021-10-17T15:28:20.147 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS-8 - Base 51 MB/s | 7.5 MB 00:00 2021-10-17T15:28:21.218 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS-8 - Extras 165 kB/s | 10 kB 00:00 2021-10-17T15:28:21.384 INFO:teuthology.orchestra.run.smithi140.stdout:CentOS-8 - PowerTools 22 MB/s | 2.4 MB 00:00 2021-10-17T15:28:24.036 INFO:teuthology.orchestra.run.smithi140.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2021-10-17T15:28:24.990 INFO:teuthology.orchestra.run.smithi140.stdout:Extra Packages for Enterprise Linux 12 MB/s | 10 MB 00:00 2021-10-17T15:28:27.122 INFO:teuthology.orchestra.run.smithi140.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2021-10-17T15:28:28.536 INFO:teuthology.orchestra.run.smithi140.stdout:Package kernel-4.18.0-193.el8.x86_64 is already installed. 2021-10-17T15:28:28.537 INFO:teuthology.orchestra.run.smithi140.stdout:Package kernel-4.18.0-305.12.1.el8_4.x86_64 is already installed. 2021-10-17T15:28:28.783 INFO:teuthology.orchestra.run.smithi140.stdout:Dependencies resolved. 2021-10-17T15:28:28.784 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2021-10-17T15:28:28.784 INFO:teuthology.orchestra.run.smithi140.stdout: Package Arch Version Repository Size 2021-10-17T15:28:28.784 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2021-10-17T15:28:28.785 INFO:teuthology.orchestra.run.smithi140.stdout:Installing: 2021-10-17T15:28:28.785 INFO:teuthology.orchestra.run.smithi140.stdout: kernel x86_64 4.18.0-305.19.1.el8_4 CentOS-Base 5.9 M 2021-10-17T15:28:28.785 INFO:teuthology.orchestra.run.smithi140.stdout:Installing dependencies: 2021-10-17T15:28:28.785 INFO:teuthology.orchestra.run.smithi140.stdout: kernel-core x86_64 4.18.0-305.19.1.el8_4 CentOS-Base 36 M 2021-10-17T15:28:28.786 INFO:teuthology.orchestra.run.smithi140.stdout: kernel-modules x86_64 4.18.0-305.19.1.el8_4 CentOS-Base 28 M 2021-10-17T15:28:28.786 INFO:teuthology.orchestra.run.smithi140.stdout: 2021-10-17T15:28:28.786 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction Summary 2021-10-17T15:28:28.786 INFO:teuthology.orchestra.run.smithi140.stdout:================================================================================ 2021-10-17T15:28:28.786 INFO:teuthology.orchestra.run.smithi140.stdout:Install 3 Packages 2021-10-17T15:28:28.787 INFO:teuthology.orchestra.run.smithi140.stdout: 2021-10-17T15:28:28.787 INFO:teuthology.orchestra.run.smithi140.stdout:Total download size: 70 M 2021-10-17T15:28:28.788 INFO:teuthology.orchestra.run.smithi140.stdout:Installed size: 88 M 2021-10-17T15:28:28.788 INFO:teuthology.orchestra.run.smithi140.stdout:Downloading Packages: 2021-10-17T15:28:28.993 INFO:teuthology.orchestra.run.smithi140.stdout:(1/3): kernel-4.18.0-305.19.1.el8_4.x86_64.rpm 29 MB/s | 5.9 MB 00:00 2021-10-17T15:28:29.584 INFO:teuthology.orchestra.run.smithi140.stdout:(2/3): kernel-core-4.18.0-305.19.1.el8_4.x86_64 45 MB/s | 36 MB 00:00 2021-10-17T15:28:29.882 INFO:teuthology.orchestra.run.smithi140.stdout:(3/3): kernel-modules-4.18.0-305.19.1.el8_4.x86 26 MB/s | 28 MB 00:01 2021-10-17T15:28:29.884 INFO:teuthology.orchestra.run.smithi140.stdout:-------------------------------------------------------------------------------- 2021-10-17T15:28:29.885 INFO:teuthology.orchestra.run.smithi140.stdout:Total 63 MB/s | 70 MB 00:01 2021-10-17T15:28:29.996 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction check 2021-10-17T15:28:30.291 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction check succeeded. 2021-10-17T15:28:30.291 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction test 2021-10-17T15:28:31.469 INFO:teuthology.orchestra.run.smithi140.stdout:Transaction test succeeded. 2021-10-17T15:28:31.498 INFO:teuthology.orchestra.run.smithi140.stdout:Running transaction 2021-10-17T15:28:33.888 INFO:teuthology.orchestra.run.smithi140.stdout: Preparing : 1/1 2021-10-17T15:28:34.443 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : kernel-core-4.18.0-305.19.1.el8_4.x86_64 1/3 2021-10-17T15:28:37.347 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: kernel-core-4.18.0-305.19.1.el8_4.x86_64 1/3 2021-10-17T15:28:37.714 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : kernel-modules-4.18.0-305.19.1.el8_4.x86_64 2/3 2021-10-17T15:28:40.689 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: kernel-modules-4.18.0-305.19.1.el8_4.x86_64 2/3 2021-10-17T15:28:40.876 INFO:teuthology.orchestra.run.smithi140.stdout: Installing : kernel-4.18.0-305.19.1.el8_4.x86_64 3/3 2021-10-17T15:44:41.419 ERROR:paramiko.transport:Socket exception: No route to host (113) 2021-10-17T15:44:41.449 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-10-17T15:44:41.450 INFO:teuthology.orchestra.run.smithi140.stdout: Running scriptlet: kernel-core-4.18.0-305.19.1.el8_4.x86_64 3/3 2021-10-17T15:44:41.450 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/task/kernel.py", line 1255, in task version = need_to_install_distro(role_remote, role_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/task/kernel.py", line 761, in need_to_install_distro 'sudo yum install -y kernel' File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/orchestra/run.py", line 175, in _raise_for_status node=self.hostname) teuthology.exceptions.ConnectionLostError: SSH connection to smithi140 was lost: 'sudo yum install -y kernel' 2021-10-17T15:44:41.541 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=fe19cf978fa348ce95686a72b1e3c94a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/task/kernel.py", line 1255, in task version = need_to_install_distro(role_remote, role_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/task/kernel.py", line 761, in need_to_install_distro 'sudo yum install -y kernel' File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/orchestra/run.py", line 175, in _raise_for_status node=self.hostname) teuthology.exceptions.ConnectionLostError: SSH connection to smithi140 was lost: 'sudo yum install -y kernel' 2021-10-17T15:44:41.543 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-10-17T15:44:41.581 INFO:teuthology.nuke:Checking targets against current locks 2021-10-17T15:44:41.747 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/run.py", line 414, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/run.py", line 271, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable