2021-05-12T22:44:42.084 INFO:root:teuthology version: 1.1.0-19220a3b 2021-05-12T22:44:42.086 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2021-05-12_22:19:02-fs-wip-yuri3-testing-2021-05-12-1004-octopus-distro-basic-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-05-12_22:19:02-fs-wip-yuri3-testing-2021-05-12-1004-octopus-distro-basic-smithi/6112036 --verbose --description fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/volumes} -- /home/teuthworker/archive/yuriw-2021-05-12_22:19:02-fs-wip-yuri3-testing-2021-05-12-1004-octopus-distro-basic-smithi/6112036/orig.config.yaml 2021-05-12T22:44:42.136 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-05-12T22:44:42.198 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2021-05-12_22:19:02-fs-wip-yuri3-testing-2021-05-12-1004-octopus-distro-basic-smithi/6112036 branch: wip-yuri3-testing-2021-05-12-1004-octopus description: fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/volumes} email: ceph-qa@ceph.io first_in_suite: false job_id: '6112036' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2021-05-12_22:19:02-fs-wip-yuri3-testing-2021-05-12-1004-octopus-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: rhel os_version: '8.2' overrides: admin_socket: branch: wip-yuri3-testing-2021-05-12-1004-octopus ceph: cephfs_ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 15m rados osd op timeout: 15m mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug client: 10 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 19aef0c0fbf48152ec623e79e93179a9d29497b9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 19aef0c0fbf48152ec623e79e93179a9d29497b9 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri3-testing-2021-05-12-1004-octopus sha1: 19aef0c0fbf48152ec623e79e93179a9d29497b9 owner: scheduled_yuriw@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.1 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 - client.2 - client.3 sha1: 19aef0c0fbf48152ec623e79e93179a9d29497b9 sleep_before_teardown: 0 suite: fs suite_branch: wip-yuri3-testing-2021-05-12-1004-octopus suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_19aef0c0fbf48152ec623e79e93179a9d29497b9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 19aef0c0fbf48152ec623e79e93179a9d29497b9 targets: smithi063.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxm9Jd35f1A64NRAOpYzRSC7/XpNjijAKlnMP1fowOZS4S/ximKHCR7bk2v8dEma1iWswOJA/Q1bf4uk0QR42faIjWs6Hfva/rZFkTgnjMdMKjFarzb8zFStJKG2UFSqgvPlkS1zHpDUxtedGtMIj0i63r6wIVC/0VoFaYDhSh/CYrcAWvOcz9kFDNUZ72168WE5tgZT95DDmxqee/ixG9Q4WzgxjJgcqXKpUWgeUJH1lh+rOKe95izYpu3y/8yDyYC71gC4Cjj/ndWzaqNVmTEyizgB/5spzRaY6jkHewJlXGSwU/LOJVAtk3ZattCQQ45/nlLLZvBXoFEjlpE6BibfeynHpAA21xM6QsiH6TL4duyhZEHCn0w6qEp8lFiAsPoYGHBiiEZ0848w4kaS1917mCogku2gWqnngLgXUyecftO/fkCGtTB825k6WtqvmgjHU3ZhSPLcR6BwL1Z+NgaKwMLCIJAPEOTVKESjL/XHI+iczSeBIPuYoDSpdjTYc= smithi122.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVHESfKmhjSi6+EUCnvYvzDBVDVvhRJtGDKNFrkz13grtQIBdAfihjKpdSGAqghX/coUaDVXljR9H2E2fmvH//h2DGuKP4mGAk3g09JGWjqoPfWjR95qnuQux/qGaQMmbZeDQNzYR391/cv6qfwC/tvOXF1xutWAUIiRf+j1jj56wqYvKrwcpBgRvQQPOfASknw6rMDjdQMhLNTgRO+vaANw6u4sYyTjcRPxEaNjhcgd9m8ryYkKbixP9GLVOAAEaetMug2Z/4rtrS2LIt5sOqn4240EduR7aGCnx+33c/PUzWctTDXgvkJiqkZOObhSVYOhECZGJef+kBXuEARdb6zJs1FmlN5elaFc62rAeKsdG6vQ+yopTOwf6pz+X4WsZzXd6Q8FW/rz3Ws6ZsB2BpCWKv3AO9IsWv21FwPahrbobPsCEt09xmpobrn9r0nbRrq0Q/PxgkrKeahcFk11Dyxt/Jsw0SWHblRGrfFcOlbZ9NxvhiOmpLbT/+XJwqvb0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes teuthology_branch: master teuthology_sha1: 19220a3bd6e252c6e8260827019668a766d85490 timestamp: 2021-05-12_22:19:02 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.8274 2021-05-12T22:44:42.198 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_19aef0c0fbf48152ec623e79e93179a9d29497b9/qa; will attempt to use it 2021-05-12T22:44:42.199 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_19aef0c0fbf48152ec623e79e93179a9d29497b9/qa/tasks 2021-05-12T22:44:42.199 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-05-12T22:44:42.202 INFO:teuthology.task.internal:Checking packages... 2021-05-12T22:44:42.224 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash '19aef0c0fbf48152ec623e79e93179a9d29497b9' 2021-05-12T22:44:42.225 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-05-12T22:44:42.225 INFO:teuthology.packaging:ref: None 2021-05-12T22:44:42.226 INFO:teuthology.packaging:tag: None 2021-05-12T22:44:42.226 INFO:teuthology.packaging:branch: wip-yuri3-testing-2021-05-12-1004-octopus 2021-05-12T22:44:42.226 INFO:teuthology.packaging:sha1: 19aef0c0fbf48152ec623e79e93179a9d29497b9 2021-05-12T22:44:42.227 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri3-testing-2021-05-12-1004-octopus 2021-05-12T22:44:42.381 INFO:teuthology.task.internal:Found packages for ceph version 15.2.11-307.g19aef0c0 2021-05-12T22:44:42.382 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-05-12T22:44:42.392 INFO:teuthology.task.internal:no buildpackages task found 2021-05-12T22:44:42.393 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-05-12T22:44:42.442 INFO:teuthology.task.internal:Saving configuration 2021-05-12T22:44:42.459 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-05-12T22:44:42.483 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-12T22:44:42.500 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-05-12 22:39:52.765387', 'description': '/home/teuthworker/archive/yuriw-2021-05-12_22:19:02-fs-wip-yuri3-testing-2021-05-12-1004-octopus-distro-basic-smithi/6112036', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGBfM6DfAUnczQjsb1FteT+UIe3OJcv/Nw/CEIIDkGZVWM11K4X5v+m2x5CEJfleQcuZdj+d3osFoMgAsYIbUC8=', 'os_version': '8.2', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi063.front.sepia.ceph.com'} 2021-05-12T22:44:42.515 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-05-12 22:39:52.765366', 'description': '/home/teuthworker/archive/yuriw-2021-05-12_22:19:02-fs-wip-yuri3-testing-2021-05-12-1004-octopus-distro-basic-smithi/6112036', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIvppDEZOfCy/i4zRwD7T3+HyysP7BPZaBawLP7lFmZ0oZIZQi5FF3MLCVHCOHLr1MDDUVlH853ASTa6p16jmNE=', 'os_version': '8.2', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi122.front.sepia.ceph.com'} 2021-05-12T22:44:42.515 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-05-12T22:44:42.543 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.1'] 2021-05-12T22:44:42.544 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.2', 'client.3'] 2021-05-12T22:44:42.544 INFO:teuthology.run_tasks:Running task console_log... 2021-05-12T22:44:42.664 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0f25e18730>, signals=[15]) 2021-05-12T22:44:42.665 INFO:teuthology.run_tasks:Running task internal.connect... 2021-05-12T22:44:42.676 INFO:teuthology.task.internal:Opening connections... 2021-05-12T22:44:42.677 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2021-05-12T22:44:42.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T22:44:42.750 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2021-05-12T22:44:42.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T22:44:42.823 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-05-12T22:44:42.831 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2021-05-12T22:44:42.878 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2021-05-12T22:44:42.879 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2021-05-12T22:44:42.934 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="Red Hat Enterprise Linux" 2021-05-12T22:44:42.934 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="8.2 (Ootpa)" 2021-05-12T22:44:42.934 INFO:teuthology.orchestra.run.smithi063.stdout:ID="rhel" 2021-05-12T22:44:42.935 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE="fedora" 2021-05-12T22:44:42.935 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="8.2" 2021-05-12T22:44:42.935 INFO:teuthology.orchestra.run.smithi063.stdout:PLATFORM_ID="platform:el8" 2021-05-12T22:44:42.935 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.2 (Ootpa)" 2021-05-12T22:44:42.936 INFO:teuthology.orchestra.run.smithi063.stdout:ANSI_COLOR="0;31" 2021-05-12T22:44:42.936 INFO:teuthology.orchestra.run.smithi063.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.2:GA" 2021-05-12T22:44:42.936 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://www.redhat.com/" 2021-05-12T22:44:42.936 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-05-12T22:44:42.937 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-05-12T22:44:42.937 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-05-12T22:44:42.937 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.2 2021-05-12T22:44:42.937 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-05-12T22:44:42.938 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.2" 2021-05-12T22:44:42.938 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2021-05-12T22:44:42.954 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2021-05-12T22:44:42.970 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2021-05-12T22:44:42.970 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2021-05-12T22:44:43.024 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="Red Hat Enterprise Linux" 2021-05-12T22:44:43.024 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="8.2 (Ootpa)" 2021-05-12T22:44:43.024 INFO:teuthology.orchestra.run.smithi122.stdout:ID="rhel" 2021-05-12T22:44:43.024 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="fedora" 2021-05-12T22:44:43.025 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="8.2" 2021-05-12T22:44:43.025 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el8" 2021-05-12T22:44:43.025 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.2 (Ootpa)" 2021-05-12T22:44:43.025 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2021-05-12T22:44:43.025 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.2:GA" 2021-05-12T22:44:43.026 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://www.redhat.com/" 2021-05-12T22:44:43.026 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-05-12T22:44:43.026 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-05-12T22:44:43.026 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-05-12T22:44:43.027 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.2 2021-05-12T22:44:43.027 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-05-12T22:44:43.027 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.2" 2021-05-12T22:44:43.027 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2021-05-12T22:44:43.041 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-05-12T22:44:43.056 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-05-12T22:44:43.100 INFO:teuthology.task.internal:Checking for old test directory... 2021-05-12T22:44:43.101 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2021-05-12T22:44:43.104 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2021-05-12T22:44:43.118 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-05-12T22:44:43.148 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-05-12T22:44:43.148 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2021-05-12T22:44:43.158 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2021-05-12T22:44:43.182 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-05-12T22:44:43.242 INFO:teuthology.run_tasks:Running task kernel... 2021-05-12T22:44:43.266 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'mds.a': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mds.b': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.2': {'kdb': True, 'sha1': 'distro'}, 'client.3': {'kdb': True, 'sha1': 'distro'}} 2021-05-12T22:44:43.267 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-05-12T22:44:43.267 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2021-05-12T22:44:43.282 INFO:teuthology.orchestra.run.smithi063.stdout:4.18.0-240.15.1.el8_3.x86_64 2021-05-12T22:44:43.283 INFO:teuthology.task.kernel:Running kernel on smithi063: 4.18.0-240.15.1.el8_3.x86_64 2021-05-12T22:44:43.283 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum install -y kernel 2021-05-12T22:44:45.049 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2021-05-12T22:44:45.049 INFO:teuthology.orchestra.run.smithi063.stdout:Unable to read consumer identity 2021-05-12T22:44:45.144 INFO:teuthology.orchestra.run.smithi063.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-05-12T22:44:47.075 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 15 MB/s | 9.4 MB 00:00 2021-05-12T22:44:49.077 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2021-05-12T22:44:49.724 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-4.18.0-193.el8.x86_64 is already installed. 2021-05-12T22:44:49.725 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-4.18.0-240.15.1.el8_3.x86_64 is already installed. 2021-05-12T22:44:49.843 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2021-05-12T22:44:49.856 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2021-05-12T22:44:49.856 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2021-05-12T22:44:49.961 DEBUG:teuthology.orchestra.run.smithi063:> echo no | sudo yum reinstall kernel || true 2021-05-12T22:44:50.312 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2021-05-12T22:44:50.313 INFO:teuthology.orchestra.run.smithi063.stdout:Unable to read consumer identity 2021-05-12T22:44:50.356 INFO:teuthology.orchestra.run.smithi063.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-05-12T22:44:50.553 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:01 ago on Wed 12 May 2021 10:44:49 PM UTC. 2021-05-12T22:44:50.686 INFO:teuthology.orchestra.run.smithi063.stdout:Installed package kernel-4.18.0-193.el8.x86_64 (from anaconda) not available. 2021-05-12T22:44:50.687 INFO:teuthology.orchestra.run.smithi063.stdout:Installed package kernel-4.18.0-240.15.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2021-05-12T22:44:50.721 INFO:teuthology.orchestra.run.smithi063.stderr:Error: No packages marked for reinstall. 2021-05-12T22:44:50.765 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum reinstall -y kernel || true 2021-05-12T22:44:51.117 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2021-05-12T22:44:51.117 INFO:teuthology.orchestra.run.smithi063.stdout:Unable to read consumer identity 2021-05-12T22:44:51.160 INFO:teuthology.orchestra.run.smithi063.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-05-12T22:44:51.352 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:02 ago on Wed 12 May 2021 10:44:49 PM UTC. 2021-05-12T22:44:51.476 INFO:teuthology.orchestra.run.smithi063.stdout:Installed package kernel-4.18.0-193.el8.x86_64 (from anaconda) not available. 2021-05-12T22:44:51.476 INFO:teuthology.orchestra.run.smithi063.stdout:Installed package kernel-4.18.0-240.15.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2021-05-12T22:44:51.502 INFO:teuthology.orchestra.run.smithi063.stderr:Error: No packages marked for reinstall. 2021-05-12T22:44:51.546 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q kernel --last | head -n 1 2021-05-12T22:44:51.629 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-4.18.0-240.15.1.el8_3.x86_64 Sun 28 Feb 2021 12:11:44 PM UTC 2021-05-12T22:44:51.631 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.15.1.el8_3.x86_64 2021-05-12T22:44:51.631 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-05-12T22:44:51.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T22:44:51.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T22:45:51.633 DEBUG:teuthology.orchestra.remote:timed out 2021-05-12T22:45:51.635 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/task/kernel.py", line 1263, in task version = need_to_install_distro(role_remote) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/task/kernel.py", line 748, in need_to_install_distro current = get_version_of_running_kernel(remote) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/task/kernel.py", line 724, in get_version_of_running_kernel uname_r = remote.sh("uname -r").strip() File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi122 was lost: 'uname -r' 2021-05-12T22:45:51.722 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f8b1647edc86427ba8dc68c8b147b158 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/task/kernel.py", line 1263, in task version = need_to_install_distro(role_remote) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/task/kernel.py", line 748, in need_to_install_distro current = get_version_of_running_kernel(remote) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/task/kernel.py", line 724, in get_version_of_running_kernel uname_r = remote.sh("uname -r").strip() File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi122 was lost: 'uname -r' 2021-05-12T22:45:51.725 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-05-12T22:45:51.856 INFO:teuthology.nuke:Checking targets against current locks 2021-05-12T22:45:51.949 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/run.py", line 412, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/run.py", line 269, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable