2021-04-09T19:04:41.022 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-09T19:04:41.024 DEBUG:teuthology.run:Teuthology command: teuthology --description multimds/basic/{0-supported-random-distro$/{rhel_7} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress} --verbose --archive /home/teuthworker/archive/yuriw-2021-04-09_18:42:12-multimds-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi/6033021 --owner scheduled_yuriw@teuthology --name yuriw-2021-04-09_18:42:12-multimds-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi -- /home/teuthworker/archive/yuriw-2021-04-09_18:42:12-multimds-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi/6033021/orig.config.yaml 2021-04-09T19:04:41.067 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-04-09T19:04:41.141 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2021-04-09_18:42:12-multimds-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi/6033021 branch: wip-yuri2-testing-2021-04-09-0846-nautilus description: multimds/basic/{0-supported-random-distro$/{rhel_7} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress} email: ceph-qa@ceph.io first_in_suite: false job_id: '6033021' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2021-04-09_18:42:12-multimds-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: rhel os_version: '7.9' overrides: admin_socket: branch: wip-yuri2-testing-2021-04-09-0846-nautilus ceph: cephfs: max_mds: 3 session_timeout: 300 cephfs_ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 fuse default permissions: false mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(TOO_FEW_PGS\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: d23cd2d5f486487b0b80655c4e33344f0f97b82b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: d23cd2d5f486487b0b80655c4e33344f0f97b82b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri2-testing-2021-04-09-0846-nautilus sha1: d23cd2d5f486487b0b80655c4e33344f0f97b82b owner: scheduled_yuriw@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mgr.x - mds.b - mds.c - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 - client.1 sha1: d23cd2d5f486487b0b80655c4e33344f0f97b82b sleep_before_teardown: 0 suite: multimds suite_branch: wip-yuri2-testing-2021-04-09-0846-nautilus suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d23cd2d5f486487b0b80655c4e33344f0f97b82b/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d23cd2d5f486487b0b80655c4e33344f0f97b82b targets: smithi032.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtHdXDSEI9AXfCJ0qN3Op96ZjXv6yLNykliQozMpQAaAbOnPPU7xxuOcMQlmbRJOrbo1A9P5RMjq9z7HAsYnCxCUGSxYASGO6jUXUNP4xBhXHGFmd5arQbaVmsvdWiEHo3Q4fmYQIS2X2qW67Fs68PGRyApknCub4aJgJGzFZsmdBXtfwLCcCsqgFW88kCGF8N5XIyT1f9AyrMFFWkwPdrqoCFKK5SKuz3SHvT7SC27aBt8dgCKLd0bBkc29ocqkMvtuBkSQYIT3uh/B/3PTyQd6T/rwOpMOCsxXjk+yLnvH0+5VGsjiNafeTM8Wsf2z4A4ybMyTYg5KcfThFG+GZZ smithi151.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6Eqb2KeuN7YHA8yC9Xl8T86tV2TXSMsofDc2vKlz45NllNLDKs+2FUv2GhLMmIZWdPzkXXPYDJ8aXZHFVvxtpCJ+LxZXPzG4Nusy3GnPxiT9Kqz8P4F6kjo+MSSXpd1saCkNSkwxi/2+1eZMFZ/j7IG1k7kaOBwcwTdaGati4gQRVpikE5Vi4a4PGKFmjYoXUR8h1REZFdfddLNZrDcfhZvDP6reJSj3bdh1VgfivDZMxCAgpCoaQakcexEeMUsDSKkFhCpb2bmE3sZI/XUcC5x9bZEf5Isk80Wb0bEYHfqnlyME0MKa+8m0AZSD/KQ6j2zZrGHwsuDILGMFxsNWf smithi183.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD3zfG3moNIYU+WKOqY1dHg6HluXn9+ne4j9ELv+K2k/nFv4z0evbYzPKhyk7SHE31zt2ajwyyMeW2ZI2qZfvk+WfM+p+/9PffbfUCAKnfYSnc9mPEIuVfzFMfyzLe866kQ9VUhx8V4xZunXQ4JgTOBTj6Y54fndeWSrmNSbFkJ76UiyA4dV+v56k/CsmZ9YWOyDahGx7WOCLyUGJq6sHNFjSVx08jLYh4I+cdVOO34c5PCIHsDUgvt2RINrFzKwjO8sFVUuycfw3NMemcdvK+yz3freNOYwy8ukC6KcAelc9+9tMLrDT2SmOuSfHFiRk93v8LYT04CoZu8TzofaWlt tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - ceph: null - ceph-fuse: null - check-counter: counters: mds: - mds.exported - mds.imported - workunit: clients: all: - suites/fsstress.sh timeout: 6h teuthology_branch: master teuthology_sha1: 85d61eae4759f46ce21e9a37cd816a7a1a66c9d5 timestamp: 2021-04-09_18:42:12 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23935 2021-04-09T19:04:41.142 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d23cd2d5f486487b0b80655c4e33344f0f97b82b/qa; will attempt to use it 2021-04-09T19:04:41.142 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d23cd2d5f486487b0b80655c4e33344f0f97b82b/qa/tasks 2021-04-09T19:04:41.143 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-04-09T19:04:41.145 INFO:teuthology.task.internal:Checking packages... 2021-04-09T19:04:41.166 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash 'd23cd2d5f486487b0b80655c4e33344f0f97b82b' 2021-04-09T19:04:41.167 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-04-09T19:04:41.168 INFO:teuthology.packaging:ref: None 2021-04-09T19:04:41.168 INFO:teuthology.packaging:tag: None 2021-04-09T19:04:41.169 INFO:teuthology.packaging:branch: wip-yuri2-testing-2021-04-09-0846-nautilus 2021-04-09T19:04:41.169 INFO:teuthology.packaging:sha1: d23cd2d5f486487b0b80655c4e33344f0f97b82b 2021-04-09T19:04:41.170 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-yuri2-testing-2021-04-09-0846-nautilus 2021-04-09T19:04:41.321 INFO:teuthology.task.internal:Found packages for ceph version 14.2.19-116.gd23cd2d5 2021-04-09T19:04:41.322 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-04-09T19:04:41.333 INFO:teuthology.task.internal:no buildpackages task found 2021-04-09T19:04:41.333 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-04-09T19:04:41.374 INFO:teuthology.task.internal:Saving configuration 2021-04-09T19:04:41.391 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-04-09T19:04:41.417 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-09T19:04:41.435 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-04-09 18:59:15.229207', 'description': '/home/teuthworker/archive/yuriw-2021-04-09_18:42:12-multimds-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi/6033021', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEhVnZHQHv7Dd4fO5DkG4POXFp+aGtdENLeSAj6FHDsotIeXVSB21Iwvk5Kx4YQiuWtz4+KIhFb7WCBngVgFWVU=', 'os_version': '7.9', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi032.front.sepia.ceph.com'} 2021-04-09T19:04:41.450 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-04-09 18:59:15.229245', 'description': '/home/teuthworker/archive/yuriw-2021-04-09_18:42:12-multimds-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi/6033021', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFOUaSihLvWf0hjU8+OHVY5/6IFxZEMa1UZq721cEEfYB/HkA1lqWUNnXRuTMJZgulXGqIqAqLrlkxlm6jTk7Tg=', 'os_version': '7.9', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi151.front.sepia.ceph.com'} 2021-04-09T19:04:41.467 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-04-09 18:59:15.229229', 'description': '/home/teuthworker/archive/yuriw-2021-04-09_18:42:12-multimds-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi/6033021', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJTNNfDCxL93Ov6loPZUWEf2vq94dQe/yOM/xsSvRVuURyU0sQvMVjjS7jzxNOljwWAwfBin+tCHt4Ghp7dUzGc=', 'os_version': '7.9', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi183.front.sepia.ceph.com'} 2021-04-09T19:04:41.467 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-04-09T19:04:41.478 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2021-04-09T19:04:41.479 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'mds.b', 'mds.c', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-04-09T19:04:41.479 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['client.0', 'client.1'] 2021-04-09T19:04:41.479 INFO:teuthology.run_tasks:Running task console_log... 2021-04-09T19:04:41.606 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb8e4735950>, signals=[15]) 2021-04-09T19:04:41.607 INFO:teuthology.run_tasks:Running task internal.connect... 2021-04-09T19:04:41.617 INFO:teuthology.task.internal:Opening connections... 2021-04-09T19:04:41.617 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2021-04-09T19:04:41.619 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-09T19:04:41.695 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2021-04-09T19:04:41.696 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-09T19:04:41.769 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2021-04-09T19:04:41.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-09T19:04:41.843 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-04-09T19:04:41.853 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2021-04-09T19:04:41.914 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2021-04-09T19:04:41.915 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2021-04-09T19:04:41.957 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="Red Hat Enterprise Linux Server" 2021-04-09T19:04:41.957 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="7.9 (Maipo)" 2021-04-09T19:04:41.957 INFO:teuthology.orchestra.run.smithi032.stdout:ID="rhel" 2021-04-09T19:04:41.958 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="fedora" 2021-04-09T19:04:41.958 INFO:teuthology.orchestra.run.smithi032.stdout:VARIANT="Server" 2021-04-09T19:04:41.958 INFO:teuthology.orchestra.run.smithi032.stdout:VARIANT_ID="server" 2021-04-09T19:04:41.958 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="7.9" 2021-04-09T19:04:41.959 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="Employee SKU" 2021-04-09T19:04:41.959 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2021-04-09T19:04:41.959 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server" 2021-04-09T19:04:41.959 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://www.redhat.com/" 2021-04-09T19:04:41.959 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-04-09T19:04:41.960 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T19:04:41.960 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2021-04-09T19:04:41.960 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.9 2021-04-09T19:04:41.960 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-04-09T19:04:41.961 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.9" 2021-04-09T19:04:41.961 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2021-04-09T19:04:41.983 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2021-04-09T19:04:42.044 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2021-04-09T19:04:42.045 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2021-04-09T19:04:42.086 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="Red Hat Enterprise Linux Server" 2021-04-09T19:04:42.087 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="7.9 (Maipo)" 2021-04-09T19:04:42.087 INFO:teuthology.orchestra.run.smithi151.stdout:ID="rhel" 2021-04-09T19:04:42.087 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="fedora" 2021-04-09T19:04:42.088 INFO:teuthology.orchestra.run.smithi151.stdout:VARIANT="Server" 2021-04-09T19:04:42.088 INFO:teuthology.orchestra.run.smithi151.stdout:VARIANT_ID="server" 2021-04-09T19:04:42.088 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="7.9" 2021-04-09T19:04:42.088 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="Employee SKU" 2021-04-09T19:04:42.089 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2021-04-09T19:04:42.089 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server" 2021-04-09T19:04:42.089 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://www.redhat.com/" 2021-04-09T19:04:42.089 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-04-09T19:04:42.090 INFO:teuthology.orchestra.run.smithi151.stdout: 2021-04-09T19:04:42.090 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2021-04-09T19:04:42.090 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.9 2021-04-09T19:04:42.090 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-04-09T19:04:42.091 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.9" 2021-04-09T19:04:42.091 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2021-04-09T19:04:42.109 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2021-04-09T19:04:42.168 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2021-04-09T19:04:42.169 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2021-04-09T19:04:42.210 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="Red Hat Enterprise Linux Server" 2021-04-09T19:04:42.210 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="7.9 (Maipo)" 2021-04-09T19:04:42.211 INFO:teuthology.orchestra.run.smithi183.stdout:ID="rhel" 2021-04-09T19:04:42.211 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="fedora" 2021-04-09T19:04:42.211 INFO:teuthology.orchestra.run.smithi183.stdout:VARIANT="Server" 2021-04-09T19:04:42.212 INFO:teuthology.orchestra.run.smithi183.stdout:VARIANT_ID="server" 2021-04-09T19:04:42.212 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="7.9" 2021-04-09T19:04:42.213 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="Employee SKU" 2021-04-09T19:04:42.213 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2021-04-09T19:04:42.214 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server" 2021-04-09T19:04:42.214 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://www.redhat.com/" 2021-04-09T19:04:42.214 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-04-09T19:04:42.215 INFO:teuthology.orchestra.run.smithi183.stdout: 2021-04-09T19:04:42.215 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2021-04-09T19:04:42.216 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.9 2021-04-09T19:04:42.216 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-04-09T19:04:42.216 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.9" 2021-04-09T19:04:42.217 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2021-04-09T19:04:42.234 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-04-09T19:04:42.249 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-04-09T19:04:42.300 INFO:teuthology.task.internal:Checking for old test directory... 2021-04-09T19:04:42.301 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2021-04-09T19:04:42.305 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2021-04-09T19:04:42.309 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2021-04-09T19:04:42.348 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-04-09T19:04:42.358 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-04-09T19:04:42.359 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2021-04-09T19:04:42.387 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2021-04-09T19:04:42.391 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2021-04-09T19:04:42.438 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-04-09T19:04:42.502 INFO:teuthology.run_tasks:Running task kernel... 2021-04-09T19:04:42.524 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'mds.a': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mds.b': {'kdb': True, 'sha1': 'distro'}, 'mds.c': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}} 2021-04-09T19:04:42.525 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-04-09T19:04:42.525 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2021-04-09T19:04:42.568 INFO:teuthology.orchestra.run.smithi032.stdout:3.10.0-1160.24.1.el7.x86_64 2021-04-09T19:04:42.569 INFO:teuthology.task.kernel:Running kernel on smithi032: 3.10.0-1160.24.1.el7.x86_64 2021-04-09T19:04:42.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum install -y kernel 2021-04-09T19:04:43.705 INFO:teuthology.orchestra.run.smithi032.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-04-09T19:04:43.706 INFO:teuthology.orchestra.run.smithi032.stdout: : disabled-repos, subscription-manager 2021-04-09T19:04:43.946 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T19:04:43.947 INFO:teuthology.orchestra.run.smithi032.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-04-09T19:04:43.947 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T19:04:43.976 INFO:teuthology.orchestra.run.smithi032.stdout:Loading mirror speeds from cached hostfile 2021-04-09T19:04:43.982 INFO:teuthology.orchestra.run.smithi032.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-04-09T19:04:45.289 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do 2021-04-09T19:04:45.309 DEBUG:teuthology.orchestra.run.smithi032:> echo no | sudo yum reinstall kernel || true 2021-04-09T19:04:45.524 INFO:teuthology.orchestra.run.smithi032.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-04-09T19:04:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: : disabled-repos, subscription-manager 2021-04-09T19:04:45.595 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T19:04:45.596 INFO:teuthology.orchestra.run.smithi032.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-04-09T19:04:45.596 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T19:04:45.678 INFO:teuthology.orchestra.run.smithi032.stderr:Skipping the running kernel: kernel-3.10.0-1160.24.1.el7.x86_64 2021-04-09T19:04:45.700 INFO:teuthology.orchestra.run.smithi032.stdout:Loading mirror speeds from cached hostfile 2021-04-09T19:04:45.701 INFO:teuthology.orchestra.run.smithi032.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-04-09T19:04:45.706 INFO:teuthology.orchestra.run.smithi032.stdout:Installed package kernel-3.10.0-1160.el7.x86_64 (from anaconda) not available. 2021-04-09T19:04:45.706 INFO:teuthology.orchestra.run.smithi032.stderr:Error: Nothing to do 2021-04-09T19:04:45.723 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel --last | head -n 1 2021-04-09T19:04:45.931 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-3.10.0-1160.24.1.el7.x86_64 Thu 08 Apr 2021 06:20:34 PM UTC 2021-04-09T19:04:45.933 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1160.24.1.el7.x86_64 2021-04-09T19:04:45.933 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-04-09T19:04:45.934 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2021-04-09T19:04:45.986 INFO:teuthology.orchestra.run.smithi151.stdout:3.10.0-1160.24.1.el7.x86_64 2021-04-09T19:04:45.987 INFO:teuthology.task.kernel:Running kernel on smithi151: 3.10.0-1160.24.1.el7.x86_64 2021-04-09T19:04:45.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum install -y kernel 2021-04-09T19:04:47.198 INFO:teuthology.orchestra.run.smithi151.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-04-09T19:04:47.198 INFO:teuthology.orchestra.run.smithi151.stdout: : disabled-repos, subscription-manager 2021-04-09T19:04:47.404 INFO:teuthology.orchestra.run.smithi151.stdout: 2021-04-09T19:04:47.404 INFO:teuthology.orchestra.run.smithi151.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-04-09T19:04:47.404 INFO:teuthology.orchestra.run.smithi151.stdout: 2021-04-09T19:04:47.431 INFO:teuthology.orchestra.run.smithi151.stdout:Loading mirror speeds from cached hostfile 2021-04-09T19:04:47.436 INFO:teuthology.orchestra.run.smithi151.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-04-09T19:04:48.572 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do 2021-04-09T19:04:48.590 DEBUG:teuthology.orchestra.run.smithi151:> echo no | sudo yum reinstall kernel || true 2021-04-09T19:04:48.791 INFO:teuthology.orchestra.run.smithi151.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-04-09T19:04:48.791 INFO:teuthology.orchestra.run.smithi151.stdout: : disabled-repos, subscription-manager 2021-04-09T19:04:48.856 INFO:teuthology.orchestra.run.smithi151.stdout: 2021-04-09T19:04:48.857 INFO:teuthology.orchestra.run.smithi151.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-04-09T19:04:48.857 INFO:teuthology.orchestra.run.smithi151.stdout: 2021-04-09T19:04:48.929 INFO:teuthology.orchestra.run.smithi151.stderr:Skipping the running kernel: kernel-3.10.0-1160.24.1.el7.x86_64 2021-04-09T19:04:48.948 INFO:teuthology.orchestra.run.smithi151.stdout:Loading mirror speeds from cached hostfile 2021-04-09T19:04:48.949 INFO:teuthology.orchestra.run.smithi151.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-04-09T19:04:48.953 INFO:teuthology.orchestra.run.smithi151.stdout:Installed package kernel-3.10.0-1160.el7.x86_64 (from anaconda) not available. 2021-04-09T19:04:48.953 INFO:teuthology.orchestra.run.smithi151.stderr:Error: Nothing to do 2021-04-09T19:04:48.970 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel --last | head -n 1 2021-04-09T19:04:49.154 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-3.10.0-1160.24.1.el7.x86_64 Thu 08 Apr 2021 06:20:34 PM UTC 2021-04-09T19:04:49.156 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1160.24.1.el7.x86_64 2021-04-09T19:04:49.156 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-04-09T19:04:49.157 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2021-04-09T19:04:49.205 INFO:teuthology.orchestra.run.smithi183.stdout:3.10.0-1160.24.1.el7.x86_64 2021-04-09T19:04:49.207 INFO:teuthology.task.kernel:Running kernel on smithi183: 3.10.0-1160.24.1.el7.x86_64 2021-04-09T19:04:49.208 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2021-04-09T19:04:50.254 INFO:teuthology.orchestra.run.smithi183.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-04-09T19:04:50.254 INFO:teuthology.orchestra.run.smithi183.stdout: : disabled-repos, subscription-manager 2021-04-09T19:04:50.414 INFO:teuthology.orchestra.run.smithi183.stdout: 2021-04-09T19:04:50.414 INFO:teuthology.orchestra.run.smithi183.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-04-09T19:04:50.415 INFO:teuthology.orchestra.run.smithi183.stdout: 2021-04-09T19:04:50.444 INFO:teuthology.orchestra.run.smithi183.stdout:Loading mirror speeds from cached hostfile 2021-04-09T19:04:50.449 INFO:teuthology.orchestra.run.smithi183.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-04-09T19:04:51.718 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do 2021-04-09T19:04:51.739 DEBUG:teuthology.orchestra.run.smithi183:> echo no | sudo yum reinstall kernel || true 2021-04-09T19:04:51.936 INFO:teuthology.orchestra.run.smithi183.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-04-09T19:04:51.937 INFO:teuthology.orchestra.run.smithi183.stdout: : disabled-repos, subscription-manager 2021-04-09T19:04:52.002 INFO:teuthology.orchestra.run.smithi183.stdout: 2021-04-09T19:04:52.002 INFO:teuthology.orchestra.run.smithi183.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-04-09T19:04:52.003 INFO:teuthology.orchestra.run.smithi183.stdout: 2021-04-09T19:04:52.086 INFO:teuthology.orchestra.run.smithi183.stderr:Skipping the running kernel: kernel-3.10.0-1160.24.1.el7.x86_64 2021-04-09T19:04:52.103 INFO:teuthology.orchestra.run.smithi183.stdout:Loading mirror speeds from cached hostfile 2021-04-09T19:04:52.104 INFO:teuthology.orchestra.run.smithi183.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-04-09T19:04:52.108 INFO:teuthology.orchestra.run.smithi183.stdout:Installed package kernel-3.10.0-1160.el7.x86_64 (from anaconda) not available. 2021-04-09T19:04:52.109 INFO:teuthology.orchestra.run.smithi183.stderr:Error: Nothing to do 2021-04-09T19:04:52.126 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel --last | head -n 1 2021-04-09T19:04:52.318 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-3.10.0-1160.24.1.el7.x86_64 Thu 08 Apr 2021 06:20:34 PM UTC 2021-04-09T19:04:52.319 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1160.24.1.el7.x86_64 2021-04-09T19:04:52.320 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-04-09T19:04:52.320 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-04-09T19:04:52.320 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-04-09T19:04:52.391 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2021-04-09T19:04:52.404 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2021-04-09T19:04:52.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-09T19:04:52.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-09T19:05:52.407 DEBUG:teuthology.orchestra.remote:timed out 2021-04-09T19:05:52.409 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/task/kernel.py", line 1342, in task enable_disable_kdb(ctx, kdb) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/task/kernel.py", line 659, in enable_disable_kdb 'sudo', 'tee', '/sys/module/kgdboc/parameters/kgdboc' File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi151 was lost: 'echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc' 2021-04-09T19:05:52.498 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ff3dc6ed4c7a465f83940453a2cfa9b8 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/task/kernel.py", line 1342, in task enable_disable_kdb(ctx, kdb) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/task/kernel.py", line 659, in enable_disable_kdb 'sudo', 'tee', '/sys/module/kgdboc/parameters/kgdboc' File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi151 was lost: 'echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc' 2021-04-09T19:05:52.500 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-04-09T19:05:52.553 INFO:teuthology.nuke:Checking targets against current locks 2021-04-09T19:05:52.668 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run.py", line 412, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run.py", line 269, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable