2023-08-31T00:07:00.294 INFO:root:teuthology version: 0.0.1.dev192+gf9a8556 2023-08-31T00:07:00.296 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} --owner scheduled_pdonnell@teuthology --name pdonnell-2023-08-30_23:48:03-fs-wip-batrick-testing-20230830.165602-distro-default-smithi --verbose --archive /home/teuthworker/archive/pdonnell-2023-08-30_23:48:03-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7384993 -- /home/teuthworker/archive/pdonnell-2023-08-30_23:48:03-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7384993/orig.config.yaml 2023-08-31T00:07:00.436 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-31T00:07:00.573 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-08-30_23:48:03-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7384993 branch: wip-batrick-testing-20230830.165602 description: fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} email: pdonnell@redhat.com first_in_suite: false job_id: '7384993' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-08-30_23:48:03-fs-wip-batrick-testing-20230830.165602-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-batrick-testing-20230830.165602 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 osd: osd op complaint time: 180 conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: cc14fc8aa878d09ac353379f17aec3b92248b995 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: cc14fc8aa878d09ac353379f17aec3b92248b995 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20230830.165602 sha1: 833faa571491534dcb6bc6b25a578499081113d8 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 - - client.1 seed: 6543 sha1: cc14fc8aa878d09ac353379f17aec3b92248b995 sleep_before_teardown: 0 subset: 131/512 suite: fs suite_branch: wip-batrick-testing-20230830.165602 suite_path: /home/teuthworker/src/github.com_batrick_ceph_833faa571491534dcb6bc6b25a578499081113d8/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: 833faa571491534dcb6bc6b25a578499081113d8 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi069.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0i5iQmPuzivqVSbzzC4iOwNEpDXlz4yaF0PK78wEcEfT0t3zn6s2iCetClxes0MW1j0bJGUa7vIEbjxmsnVikZzjo8VwtPuFfV/Fl8tAu0p2Mz94IZ1GrLXcx9rMh1awaMJ03UpMFNZQEFSQ1zZg+FoaUew8LetXbEf/EhdKtOr6FbiN/OoZF18BM6DWliti8ua6UC/2d/gNjRnmpwN9tmANbOUz374ig6ssnmQ7r7lqKQkzCF2Fn5ut3xapXTHW2yy+QTY9dQ4K4V5gcGFpsWtt0ID7ThtXmTAkX1QxeXFrO6tzSCJ66QnZEUtpE3ZoQlDkbQ6yIOv0/1U1BiyeGdaNYO//z0NlGU+B49vSZoNKhOqhLQBhCJ94se9tdeJojIGhCr+pb/5mnqHlqax6FvW/xR+jZrQ5tt7gyvgYBqQuoZ9FrdZg01NiDDGO6xs2HuiynKHqPypccJGzwcSZIk1YIDdpvSjyx7c/irxWBc+phzWbeRHS0tih4aB7gdd8= smithi149.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsmlabSgyMiPIfxXlcgcXCgjQ8WAt+7xa1aPr0wbq9a705DRjZhSAnZupozXH57wBbbxdNKs7JnibXjtLvZxiFe+lXZ8LzwKCymIrIdccHuW5oyj2E9xlIQMJMJhW0y9xPRfFipVl2enhT42vqB08gzpe8JKJSfECRT+zmdwceVyUYHPdZTaOU2OOCO4nxiZj9rxK+D8buu3plLru2bJrQilBsX3CmsjZRsN6QoQPdKONLI0uqjgrTtdb3bEjDElL/Cpes4tXU74SlpIFZss9cGJ+1MFDeLtpe1ZLSxQENbPYNy3V4IQNl8Ns1yN9UfCeR0IJyz7rwiGNHzSzolV7AP5dpfNmKJhYQgt88OwMGIHPX34+omhlcoeSTeiDvBn/c3pp1COyNqItoB4rPWVM7LSvQc1fyHikyoNgGYi3Dwi5wRINwshsLnEq2Hr2UHXDtWlQ5jOpuEh232irEfzEoz9JEnMQtdPxMfcnhQEUqc9y2RmLk+JZytBwtXf/Qi1s= smithi153.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiH8Ar6mRtNJoNdBUK7BjS0tJM6kK34WUrA3Ar7S/BN9TMykVkzlZr6UXn+LhD/PlY9WM9MYVSJI68texiV7X48MDlZ/CoSS+LOJZn9GGDiuJkCufVEJT1xZVR1ZYUSNjIlyYGIGA89c57O3qvUXdSJjnCQaNufuzDS9p85mvcS4gxSEL/kj4ya66xZnSZGKTqLqQ2QoLB2M9KfueQejkoObZ96HJhLFQ7tBk5x8ZKrh5V+lJTVtGzQcgkk3oSyhteJHpf55961a1VqGqev0hOb4FXnsgtcHkemMyddwNXbWsDsGm9DoyVTg8W1L5n3NVJ1LO1mD4iECbgG+tr2NZk1F+sI4NwsbZWMU3F5XUir9wlkAi4vNb0FBth5zQMsD+wqGzOXJd5LeopcLxJrFhbecs9w2n+XVGyQ5toGPz3PhxNplHvDv8hoOKh2O+9Uuf+HOj7AUECaDBmLKlMihNMNsFE5U+rF6cK4lc6I4T2nG6V0tx6idCWzQYsfNnQJ0s= smithi177.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTVk2h5mGPdwIiEKQnV7J050bJ8mhPg6nQ3dE729R5TpHi6wZclYvnKe4HSPsfuLinh0gYLOnnhIESEYkjcR8dvoflrTxZcl9Nwj+oj1u4icS3xSlU9b8oUE4vfF0L9qaU/D6OB/fiso42hNLZFJO94Xd1/MrNY4VZH65OZgAaxysV4iMPJpj+DM7amK/Vm8FNLOWvtUgKdhGJxmW0QGePu3y2RBQC/EWBQ+K4Ru7iDhaazOHLxruNAFMlkOfQuiMWfoyYwkea7aDlWjgxeWxAon9rxIzMOo5kkCH5FWsGZgzpTN3wsEjSuK98CCFkXCswW4UCqzWK5B+Y+ZnRA0acA3rR6SXj+dprYuNAIEcb2Ug+3QKh18ArHH2vmYBhXC07CUG5o5ZsnJMlHTRf0xKDeqIkbKCkj4yx2XuQcQ45jIZzuFzQ4TMi3hWTcHGuvKbGpA+Uxxuift5xCCRSvGfJDYjTDN+8S7irpGkWJewfySs+qDqOckCtIpjASY52yRU= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - pexec: clients: - cd $TESTDIR - wget http://download.ceph.com/qa/mdtest-1.9.3.tgz - mkdir mdtest-1.9.3 - cd mdtest-1.9.3 - tar xvfz $TESTDIR/mdtest-1.9.3.tgz - rm $TESTDIR/mdtest-1.9.3.tgz - MPI_CC=mpicc make - ln -s $TESTDIR/mnt.* $TESTDIR/gmnt - ssh_keys: null - mpi: exec: $TESTDIR/mdtest-1.9.3/mdtest -d $TESTDIR/gmnt -I 20 -z 5 -b 2 -R - pexec: clients: - rm -f $TESTDIR/gmnt - rm -rf $TESTDIR/mdtest-1.9.3 - rm -rf $TESTDIR/._mdtest-1.9.3 teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: f9a8556d5e989d27b6626a0b75ef595991f83d20 timestamp: 2023-08-30_23:48:03 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1394 2023-08-31T00:07:00.573 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_833faa571491534dcb6bc6b25a578499081113d8/qa; will attempt to use it 2023-08-31T00:07:00.574 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_833faa571491534dcb6bc6b25a578499081113d8/qa/tasks 2023-08-31T00:07:00.575 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-08-31T00:07:00.580 INFO:teuthology.task.internal:Checking packages... 2023-08-31T00:07:00.614 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'cc14fc8aa878d09ac353379f17aec3b92248b995' 2023-08-31T00:07:00.614 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-08-31T00:07:00.615 INFO:teuthology.packaging:ref: None 2023-08-31T00:07:00.615 INFO:teuthology.packaging:tag: None 2023-08-31T00:07:00.615 INFO:teuthology.packaging:branch: wip-batrick-testing-20230830.165602 2023-08-31T00:07:00.616 INFO:teuthology.packaging:sha1: cc14fc8aa878d09ac353379f17aec3b92248b995 2023-08-31T00:07:00.616 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-batrick-testing-20230830.165602 2023-08-31T00:07:00.761 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5814-gcc14fc8a-1jammy 2023-08-31T00:07:00.762 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-08-31T00:07:00.797 INFO:teuthology.task.internal:no buildpackages task found 2023-08-31T00:07:00.797 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-08-31T00:07:00.811 INFO:teuthology.task.internal:Saving configuration 2023-08-31T00:07:00.834 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-08-31T00:07:00.847 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-31T00:07:00.885 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-08-30_23:48:03-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7384993', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 00:01:06.087170', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIASF7lukUicJnm+DevQs9USkddNnJEfL78gom8OJIac8'} 2023-08-31T00:07:00.923 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-08-30_23:48:03-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7384993', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 00:01:06.084369', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILhpguH/bsgYzEy8FTozsFAFF4O/LU8NRkYfnNS7FycA'} 2023-08-31T00:07:00.951 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-08-30_23:48:03-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7384993', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 00:01:06.085833', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKt/49dIE//Qtln76aOjH6I6JRrbVqtFYkCW6mVxeKPy'} 2023-08-31T00:07:00.973 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-08-30_23:48:03-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7384993', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 00:01:06.088453', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMKVP4iVrqFMDYcw4I/SuFf/LaT5YVM+5fLMsNyNa/JN'} 2023-08-31T00:07:00.973 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-08-31T00:07:00.990 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-08-31T00:07:00.990 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-08-31T00:07:00.990 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['client.0'] 2023-08-31T00:07:00.991 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['client.1'] 2023-08-31T00:07:00.991 INFO:teuthology.run_tasks:Running task console_log... 2023-08-31T00:07:01.235 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd57a3d2af0>, signals=[15]) 2023-08-31T00:07:01.237 INFO:teuthology.run_tasks:Running task internal.connect... 2023-08-31T00:07:01.277 INFO:teuthology.task.internal:Opening connections... 2023-08-31T00:07:01.277 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2023-08-31T00:07:01.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-31T00:07:01.314 ERROR:teuthology.orchestra.connection:Error authenticating with smithi069.front.sepia.ceph.com: No SSH private key found! 2023-08-31T00:07:01.316 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/task/internal/__init__.py", line 180, in connect rem.connect() File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/orchestra/remote.py", line 386, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 438, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi069.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIASF7lukUicJnm+DevQs9USkddNnJEfL78gom8OJIac8', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC0i5iQmPuzivqVSbzzC4iOwNEpDXlz4yaF0PK78wEcEfT0t3zn6s2iCetClxes0MW1j0bJGUa7vIEbjxmsnVikZzjo8VwtPuFfV/Fl8tAu0p2Mz94IZ1GrLXcx9rMh1awaMJ03UpMFNZQEFSQ1zZg+FoaUew8LetXbEf/EhdKtOr6FbiN/OoZF18BM6DWliti8ua6UC/2d/gNjRnmpwN9tmANbOUz374ig6ssnmQ7r7lqKQkzCF2Fn5ut3xapXTHW2yy+QTY9dQ4K4V5gcGFpsWtt0ID7ThtXmTAkX1QxeXFrO6tzSCJ66QnZEUtpE3ZoQlDkbQ6yIOv0/1U1BiyeGdaNYO//z0NlGU+B49vSZoNKhOqhLQBhCJ94se9tdeJojIGhCr+pb/5mnqHlqax6FvW/xR+jZrQ5tt7gyvgYBqQuoZ9FrdZg01NiDDGO6xs2HuiynKHqPypccJGzwcSZIk1YIDdpvSjyx7c/irxWBc+phzWbeRHS0tih4aB7gdd8=' 2023-08-31T00:07:01.512 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5a251e7951f84a6a8c6ff8640534876f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/task/internal/__init__.py", line 180, in connect rem.connect() File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/orchestra/remote.py", line 386, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 438, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi069.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIASF7lukUicJnm+DevQs9USkddNnJEfL78gom8OJIac8', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC0i5iQmPuzivqVSbzzC4iOwNEpDXlz4yaF0PK78wEcEfT0t3zn6s2iCetClxes0MW1j0bJGUa7vIEbjxmsnVikZzjo8VwtPuFfV/Fl8tAu0p2Mz94IZ1GrLXcx9rMh1awaMJ03UpMFNZQEFSQ1zZg+FoaUew8LetXbEf/EhdKtOr6FbiN/OoZF18BM6DWliti8ua6UC/2d/gNjRnmpwN9tmANbOUz374ig6ssnmQ7r7lqKQkzCF2Fn5ut3xapXTHW2yy+QTY9dQ4K4V5gcGFpsWtt0ID7ThtXmTAkX1QxeXFrO6tzSCJ66QnZEUtpE3ZoQlDkbQ6yIOv0/1U1BiyeGdaNYO//z0NlGU+B49vSZoNKhOqhLQBhCJ94se9tdeJojIGhCr+pb/5mnqHlqax6FvW/xR+jZrQ5tt7gyvgYBqQuoZ9FrdZg01NiDDGO6xs2HuiynKHqPypccJGzwcSZIk1YIDdpvSjyx7c/irxWBc+phzWbeRHS0tih4aB7gdd8=' 2023-08-31T00:07:01.515 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-08-31T00:07:01.585 INFO:teuthology.nuke:Checking targets against current locks 2023-08-31T00:07:01.638 DEBUG:teuthology.nuke:shortname: smithi069 2023-08-31T00:07:01.639 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-31T00:07:01.700 DEBUG:teuthology.nuke:shortname: smithi149 2023-08-31T00:07:01.702 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-31T00:07:01.717 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-08-30_23:48:03-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7384993', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 00:01:06.087170', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIASF7lukUicJnm+DevQs9USkddNnJEfL78gom8OJIac8'} 2023-08-31T00:07:01.738 DEBUG:teuthology.nuke:shortname: smithi153 2023-08-31T00:07:01.738 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-31T00:07:01.753 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-08-30_23:48:03-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7384993', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 00:01:06.084369', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILhpguH/bsgYzEy8FTozsFAFF4O/LU8NRkYfnNS7FycA'} 2023-08-31T00:07:01.779 DEBUG:teuthology.nuke:shortname: smithi177 2023-08-31T00:07:01.779 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-31T00:07:01.797 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-08-30_23:48:03-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7384993', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 00:01:06.085833', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKt/49dIE//Qtln76aOjH6I6JRrbVqtFYkCW6mVxeKPy'} 2023-08-31T00:07:01.812 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-08-30_23:48:03-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7384993', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 00:01:06.088453', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMKVP4iVrqFMDYcw4I/SuFf/LaT5YVM+5fLMsNyNa/JN'} 2023-08-31T00:07:01.841 INFO:teuthology.orchestra.console.smithi069:Power off 2023-08-31T00:07:01.842 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-31T00:07:01.888 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2023-08-31T00:07:01.889 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T00:07:01.913 INFO:teuthology.orchestra.console.smithi149:Power off 2023-08-31T00:07:01.914 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-31T00:07:01.941 INFO:teuthology.orchestra.console.smithi153:Power off 2023-08-31T00:07:01.942 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-31T00:07:01.982 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2023-08-31T00:07:01.983 DEBUG:teuthology.orchestra.console.smithi149:power off output: Chassis Power Control: Down/Off 2023-08-31T00:07:01.983 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T00:07:01.999 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2023-08-31T00:07:02.001 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T00:07:02.031 INFO:teuthology.orchestra.console.smithi177:Power off 2023-08-31T00:07:02.032 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-31T00:07:02.066 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2023-08-31T00:07:02.067 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2023-08-31T00:07:02.080 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2023-08-31T00:07:02.080 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T00:07:02.122 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2023-08-31T00:07:05.984 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T00:07:06.069 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T00:07:06.082 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T00:07:06.106 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2023-08-31T00:07:06.124 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T00:07:06.186 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2023-08-31T00:07:06.202 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2023-08-31T00:07:06.238 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2023-08-31T00:07:10.109 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T00:07:10.187 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T00:07:10.206 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T00:07:10.224 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2023-08-31T00:07:10.239 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T00:07:10.302 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2023-08-31T00:07:10.323 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2023-08-31T00:07:10.351 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2023-08-31T00:07:10.403 INFO:teuthology.orchestra.console.smithi153:Power off completed 2023-08-31T00:07:14.230 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T00:07:14.325 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T00:07:14.345 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2023-08-31T00:07:14.354 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T00:07:14.442 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is off 2023-08-31T00:07:14.449 INFO:teuthology.orchestra.console.smithi069:Power off completed 2023-08-31T00:07:14.471 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2023-08-31T00:07:14.543 INFO:teuthology.orchestra.console.smithi149:Power off completed 2023-08-31T00:07:14.573 INFO:teuthology.orchestra.console.smithi177:Power off completed 2023-08-31T00:07:14.685 INFO:teuthology.run:Summary data: description: fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} failure_reason: 'Host key for server ''smithi069.front.sepia.ceph.com'' does not match: got ''AAAAC3NzaC1lZDI1NTE5AAAAIASF7lukUicJnm+DevQs9USkddNnJEfL78gom8OJIac8'', expected ''AAAAB3NzaC1yc2EAAAADAQABAAABgQC0i5iQmPuzivqVSbzzC4iOwNEpDXlz4yaF0PK78wEcEfT0t3zn6s2iCetClxes0MW1j0bJGUa7vIEbjxmsnVikZzjo8VwtPuFfV/Fl8tAu0p2Mz94IZ1GrLXcx9rMh1awaMJ03UpMFNZQEFSQ1zZg+FoaUew8LetXbEf/EhdKtOr6FbiN/OoZF18BM6DWliti8ua6UC/2d/gNjRnmpwN9tmANbOUz374ig6ssnmQ7r7lqKQkzCF2Fn5ut3xapXTHW2yy+QTY9dQ4K4V5gcGFpsWtt0ID7ThtXmTAkX1QxeXFrO6tzSCJ66QnZEUtpE3ZoQlDkbQ6yIOv0/1U1BiyeGdaNYO//z0NlGU+B49vSZoNKhOqhLQBhCJ94se9tdeJojIGhCr+pb/5mnqHlqax6FvW/xR+jZrQ5tt7gyvgYBqQuoZ9FrdZg01NiDDGO6xs2HuiynKHqPypccJGzwcSZIk1YIDdpvSjyx7c/irxWBc+phzWbeRHS0tih4aB7gdd8=''' owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5a251e7951f84a6a8c6ff8640534876f status: fail success: false 2023-08-31T00:07:14.686 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-31T00:07:14.783 INFO:teuthology.run:FAIL