2024-04-08T21:08:14.178 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-08T21:08:14.179 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-04-02_20:16:13-rbd-main-distro-default-smithi --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-02_20:16:13-rbd-main-distro-default-smithi/7636924 --verbose --description rbd/librbd/{cache/none clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/replicated extra-conf/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_defaults} -- /home/teuthworker/archive/teuthology-2024-04-02_20:16:13-rbd-main-distro-default-smithi/7636924/orig.config.yaml 2024-04-08T21:08:14.212 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:08:14.295 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-02_20:16:13-rbd-main-distro-default-smithi/7636924 branch: main description: rbd/librbd/{cache/none clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/replicated extra-conf/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_defaults} email: ceph-qa@ceph.com first_in_suite: false job_id: '7636924' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-02_20:16:13-rbd-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: client: rbd clone copy on read: true rbd default data pool: datapool global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - is full \(reached quota - \(POOL_FULL\) sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: d7e176b41636f150fe82aaedea82def3e9af0203 owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mgr.y - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 seed: 4671 sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d sleep_before_teardown: 0 subset: 30/128 suite: rbd suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_d7e176b41636f150fe82aaedea82def3e9af0203/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d7e176b41636f150fe82aaedea82def3e9af0203 targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGNM+HgiZ38PkT4OcoRIK1jN269RStuZ1/2kw3cUi7Ha2hAUK/ZGtPo+5RhzSHeF+G1YzZfyYKvG6oZTVtLW/uo= smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEo4vujvwmon9uRFcLCVFqTlUz3mnBdsL208GCTuZWgSEeVHDcDzFnbYn4RPD1qlYi5AayFPep7hHOXkkj5kJo0= smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGYXj0xhY9pr2jGASIuufXsALbag7HE+J7hpYLFXdjSKrq4slerA8A9+WtbZ7anmI+c/kqMQwRDwVo9wDtLp7bQ= tasks: - install: null - ceph: conf: client: rbd cache: false - exec: client.0: - sudo ceph osd pool create datapool 4 - rbd pool init datapool - exec: client.0: - sudo ceph osd set-require-min-compat-client octopus - workunit: clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-04-02_20:16:13 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-08T21:08:14.295 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d7e176b41636f150fe82aaedea82def3e9af0203/qa; will attempt to use it 2024-04-08T21:08:14.296 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d7e176b41636f150fe82aaedea82def3e9af0203/qa/tasks 2024-04-08T21:08:14.296 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-08T21:08:14.298 INFO:teuthology.task.internal:Checking packages... 2024-04-08T21:08:14.324 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'e5c885fe95ee982ff7b45214daa61b1a6d09d57d' 2024-04-08T21:08:14.324 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-08T21:08:14.324 INFO:teuthology.packaging:ref: None 2024-04-08T21:08:14.324 INFO:teuthology.packaging:tag: None 2024-04-08T21:08:14.324 INFO:teuthology.packaging:branch: main 2024-04-08T21:08:14.324 INFO:teuthology.packaging:sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-08T21:08:14.325 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-04-08T21:08:14.575 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2793-gbf893680-1jammy 2024-04-08T21:08:14.577 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-08T21:08:14.584 INFO:teuthology.task.internal:no buildpackages task found 2024-04-08T21:08:14.584 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-08T21:08:14.593 INFO:teuthology.task.internal:Saving configuration 2024-04-08T21:08:14.605 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-08T21:08:14.611 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-08T21:08:14.635 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-02_20:16:13-rbd-main-distro-default-smithi/7636924', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-08 20:57:48.143104', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIzgRkk1liuVH449e5mU782HhX0DB0IASBRws/nK0vDx'} 2024-04-08T21:08:14.657 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-02_20:16:13-rbd-main-distro-default-smithi/7636924', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-08 20:57:48.141599', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEhVBJafK1Za7rFCBqZQ+2bwbSzulRkC6RjkvR4A360L'} 2024-04-08T21:08:14.681 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-02_20:16:13-rbd-main-distro-default-smithi/7636941', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-08 21:03:24.956183', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIoDjF4KzxcOtgwi72pe0Vg7iQlEV7JcM+qkXKXyyfLfRnxMPvW/D5pWTviOY/otqHIDa5hgJboBISXgTi/VgfY='} 2024-04-08T21:08:14.681 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-08T21:08:14.687 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2024-04-08T21:08:14.687 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-08T21:08:14.687 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['client.0'] 2024-04-08T21:08:14.687 INFO:teuthology.run_tasks:Running task console_log... 2024-04-08T21:08:14.801 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f28b12ce3a0>, signals=[15]) 2024-04-08T21:08:14.801 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-08T21:08:14.807 INFO:teuthology.task.internal:Opening connections... 2024-04-08T21:08:14.808 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-04-08T21:08:14.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T21:08:14.890 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-04-08T21:08:14.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T21:08:14.988 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2024-04-08T21:08:14.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T21:08:15.007 ERROR:teuthology.orchestra.connection:Error authenticating with smithi184.front.sepia.ceph.com: BadHostKeyException('smithi184.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QTMDgXXbQtretjulr0/jak/tHPyhRWBZBq79kQZGftk), PKey(alg=ECDSA, bits=256, fp=SHA256:I/Rd13Wlef7E06dLzlJD3rCWr7rP7s7sKKFrGRb0YCY)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi184.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIoDjF4KzxcOtgwi72pe0Vg7iQlEV7JcM+qkXKXyyfLfRnxMPvW/D5pWTviOY/otqHIDa5hgJboBISXgTi/VgfY=', expected 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGYXj0xhY9pr2jGASIuufXsALbag7HE+J7hpYLFXdjSKrq4slerA8A9+WtbZ7anmI+c/kqMQwRDwVo9wDtLp7bQ=' 2024-04-08T21:08:19.026 ERROR:teuthology.orchestra.connection:Error authenticating with smithi184.front.sepia.ceph.com: BadHostKeyException('smithi184.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QTMDgXXbQtretjulr0/jak/tHPyhRWBZBq79kQZGftk), PKey(alg=ECDSA, bits=256, fp=SHA256:I/Rd13Wlef7E06dLzlJD3rCWr7rP7s7sKKFrGRb0YCY)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi184.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIoDjF4KzxcOtgwi72pe0Vg7iQlEV7JcM+qkXKXyyfLfRnxMPvW/D5pWTviOY/otqHIDa5hgJboBISXgTi/VgfY=', expected 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGYXj0xhY9pr2jGASIuufXsALbag7HE+J7hpYLFXdjSKrq4slerA8A9+WtbZ7anmI+c/kqMQwRDwVo9wDtLp7bQ=' 2024-04-08T21:08:26.047 ERROR:teuthology.orchestra.connection:Error authenticating with smithi184.front.sepia.ceph.com: BadHostKeyException('smithi184.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QTMDgXXbQtretjulr0/jak/tHPyhRWBZBq79kQZGftk), PKey(alg=ECDSA, bits=256, fp=SHA256:I/Rd13Wlef7E06dLzlJD3rCWr7rP7s7sKKFrGRb0YCY)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi184.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIoDjF4KzxcOtgwi72pe0Vg7iQlEV7JcM+qkXKXyyfLfRnxMPvW/D5pWTviOY/otqHIDa5hgJboBISXgTi/VgfY=', expected 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGYXj0xhY9pr2jGASIuufXsALbag7HE+J7hpYLFXdjSKrq4slerA8A9+WtbZ7anmI+c/kqMQwRDwVo9wDtLp7bQ=' 2024-04-08T21:08:36.065 ERROR:teuthology.orchestra.connection:Error authenticating with smithi184.front.sepia.ceph.com: BadHostKeyException('smithi184.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QTMDgXXbQtretjulr0/jak/tHPyhRWBZBq79kQZGftk), PKey(alg=ECDSA, bits=256, fp=SHA256:I/Rd13Wlef7E06dLzlJD3rCWr7rP7s7sKKFrGRb0YCY)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi184.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIoDjF4KzxcOtgwi72pe0Vg7iQlEV7JcM+qkXKXyyfLfRnxMPvW/D5pWTviOY/otqHIDa5hgJboBISXgTi/VgfY=', expected 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGYXj0xhY9pr2jGASIuufXsALbag7HE+J7hpYLFXdjSKrq4slerA8A9+WtbZ7anmI+c/kqMQwRDwVo9wDtLp7bQ=' 2024-04-08T21:08:49.086 ERROR:teuthology.orchestra.connection:Error authenticating with smithi184.front.sepia.ceph.com: BadHostKeyException('smithi184.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QTMDgXXbQtretjulr0/jak/tHPyhRWBZBq79kQZGftk), PKey(alg=ECDSA, bits=256, fp=SHA256:I/Rd13Wlef7E06dLzlJD3rCWr7rP7s7sKKFrGRb0YCY)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi184.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIoDjF4KzxcOtgwi72pe0Vg7iQlEV7JcM+qkXKXyyfLfRnxMPvW/D5pWTviOY/otqHIDa5hgJboBISXgTi/VgfY=', expected 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGYXj0xhY9pr2jGASIuufXsALbag7HE+J7hpYLFXdjSKrq4slerA8A9+WtbZ7anmI+c/kqMQwRDwVo9wDtLp7bQ=' 2024-04-08T21:09:05.105 ERROR:teuthology.orchestra.connection:Error authenticating with smithi184.front.sepia.ceph.com: BadHostKeyException('smithi184.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QTMDgXXbQtretjulr0/jak/tHPyhRWBZBq79kQZGftk), PKey(alg=ECDSA, bits=256, fp=SHA256:I/Rd13Wlef7E06dLzlJD3rCWr7rP7s7sKKFrGRb0YCY)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi184.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIoDjF4KzxcOtgwi72pe0Vg7iQlEV7JcM+qkXKXyyfLfRnxMPvW/D5pWTviOY/otqHIDa5hgJboBISXgTi/VgfY=', expected 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGYXj0xhY9pr2jGASIuufXsALbag7HE+J7hpYLFXdjSKrq4slerA8A9+WtbZ7anmI+c/kqMQwRDwVo9wDtLp7bQ=' 2024-04-08T21:09:24.126 ERROR:teuthology.orchestra.connection:Error authenticating with smithi184.front.sepia.ceph.com: BadHostKeyException('smithi184.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QTMDgXXbQtretjulr0/jak/tHPyhRWBZBq79kQZGftk), PKey(alg=ECDSA, bits=256, fp=SHA256:I/Rd13Wlef7E06dLzlJD3rCWr7rP7s7sKKFrGRb0YCY)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi184.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIoDjF4KzxcOtgwi72pe0Vg7iQlEV7JcM+qkXKXyyfLfRnxMPvW/D5pWTviOY/otqHIDa5hgJboBISXgTi/VgfY=', expected 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGYXj0xhY9pr2jGASIuufXsALbag7HE+J7hpYLFXdjSKrq4slerA8A9+WtbZ7anmI+c/kqMQwRDwVo9wDtLp7bQ=' 2024-04-08T21:09:46.145 ERROR:teuthology.orchestra.connection:Error authenticating with smithi184.front.sepia.ceph.com: BadHostKeyException('smithi184.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QTMDgXXbQtretjulr0/jak/tHPyhRWBZBq79kQZGftk), PKey(alg=ECDSA, bits=256, fp=SHA256:I/Rd13Wlef7E06dLzlJD3rCWr7rP7s7sKKFrGRb0YCY)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi184.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIoDjF4KzxcOtgwi72pe0Vg7iQlEV7JcM+qkXKXyyfLfRnxMPvW/D5pWTviOY/otqHIDa5hgJboBISXgTi/VgfY=', expected 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGYXj0xhY9pr2jGASIuufXsALbag7HE+J7hpYLFXdjSKrq4slerA8A9+WtbZ7anmI+c/kqMQwRDwVo9wDtLp7bQ=' 2024-04-08T21:10:11.168 ERROR:teuthology.orchestra.connection:Error authenticating with smithi184.front.sepia.ceph.com: BadHostKeyException('smithi184.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QTMDgXXbQtretjulr0/jak/tHPyhRWBZBq79kQZGftk), PKey(alg=ECDSA, bits=256, fp=SHA256:I/Rd13Wlef7E06dLzlJD3rCWr7rP7s7sKKFrGRb0YCY)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi184.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIoDjF4KzxcOtgwi72pe0Vg7iQlEV7JcM+qkXKXyyfLfRnxMPvW/D5pWTviOY/otqHIDa5hgJboBISXgTi/VgfY=', expected 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGYXj0xhY9pr2jGASIuufXsALbag7HE+J7hpYLFXdjSKrq4slerA8A9+WtbZ7anmI+c/kqMQwRDwVo9wDtLp7bQ=' 2024-04-08T21:11:39.173 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 180, in connect rem.connect() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 387, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 386, in connect sock.connect(addr) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect self._internal_connect(address) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 640, in _internal_connect self._wait(self._write_event) File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket File "src/gevent/_hub_primitives.py", line 313, in gevent._gevent_c_hub_primitives._primitive_wait File "src/gevent/_hub_primitives.py", line 314, in gevent._gevent_c_hub_primitives._primitive_wait File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch socket.timeout: timed out 2024-04-08T21:11:39.401 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=dd706ff9c95146b88ae0540dd826c5ab Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 180, in connect rem.connect() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 387, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 386, in connect sock.connect(addr) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 590, in connect self._internal_connect(address) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 640, in _internal_connect self._wait(self._write_event) File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket File "src/gevent/_hub_primitives.py", line 313, in gevent._gevent_c_hub_primitives._primitive_wait File "src/gevent/_hub_primitives.py", line 314, in gevent._gevent_c_hub_primitives._primitive_wait File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch socket.timeout: timed out 2024-04-08T21:11:39.404 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-08T21:11:39.425 INFO:teuthology.nuke:Checking targets against current locks 2024-04-08T21:11:39.464 DEBUG:teuthology.nuke:shortname: smithi006 2024-04-08T21:11:39.465 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-08T21:11:39.486 DEBUG:teuthology.nuke:shortname: smithi090 2024-04-08T21:11:39.486 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-08T21:11:39.491 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-02_20:16:13-rbd-main-distro-default-smithi/7636924', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-08 20:57:48.143104', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIzgRkk1liuVH449e5mU782HhX0DB0IASBRws/nK0vDx'} 2024-04-08T21:11:39.508 DEBUG:teuthology.nuke:shortname: smithi184 2024-04-08T21:11:39.508 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-08T21:11:39.512 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-02_20:16:13-rbd-main-distro-default-smithi/7636924', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-08 20:57:48.141599', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEhVBJafK1Za7rFCBqZQ+2bwbSzulRkC6RjkvR4A360L'} 2024-04-08T21:11:39.525 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-02_20:16:13-rbd-main-distro-default-smithi/7636941', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-08 21:03:24.956183', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIoDjF4KzxcOtgwi72pe0Vg7iQlEV7JcM+qkXKXyyfLfRnxMPvW/D5pWTviOY/otqHIDa5hgJboBISXgTi/VgfY='} 2024-04-08T21:11:39.545 INFO:teuthology.orchestra.console.smithi006:Power off 2024-04-08T21:11:39.545 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-08T21:11:39.572 DEBUG:teuthology.orchestra.console.smithi006:power off output: Chassis Power Control: Down/Off 2024-04-08T21:11:39.572 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T21:11:39.588 INFO:teuthology.orchestra.console.smithi090:Power off 2024-04-08T21:11:39.588 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-08T21:11:39.603 INFO:teuthology.orchestra.console.smithi184:Power off 2024-04-08T21:11:39.604 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-08T21:11:39.611 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-04-08T21:11:39.612 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off 2024-04-08T21:11:39.612 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T21:11:39.625 DEBUG:teuthology.orchestra.console.smithi184:power off output: Chassis Power Control: Down/Off 2024-04-08T21:11:39.625 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T21:11:39.634 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-04-08T21:11:39.648 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-04-08T21:11:43.613 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T21:11:43.635 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T21:11:43.649 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T21:11:43.722 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-04-08T21:11:43.748 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-04-08T21:11:43.759 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is off 2024-04-08T21:11:43.860 INFO:teuthology.orchestra.console.smithi184:Power off completed 2024-04-08T21:11:47.726 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T21:11:47.748 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T21:11:47.838 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is off 2024-04-08T21:11:47.859 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off 2024-04-08T21:11:47.939 INFO:teuthology.orchestra.console.smithi006:Power off completed 2024-04-08T21:11:47.960 INFO:teuthology.orchestra.console.smithi090:Power off completed 2024-04-08T21:11:48.064 INFO:teuthology.run:Summary data: description: rbd/librbd/{cache/none clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/replicated extra-conf/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_defaults} failure_reason: timed out owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=dd706ff9c95146b88ae0540dd826c5ab status: fail success: false 2024-04-08T21:11:48.065 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:11:48.187 INFO:teuthology.run:FAIL