2024-03-28T23:27:22.907 INFO:root:teuthology version: 0.0.1.dev262+ge691533
2024-03-28T23:27:22.907 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-28T23:27:23.057 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fcdc1360f40>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fcdc23d3c10>, signals=[15])
2024-03-28T23:27:23.058 INFO:teuthology.lock.ops:Start node 'smithi092.front.sepia.ceph.com' reimaging
2024-03-28T23:27:23.058 INFO:teuthology.lock.ops:Updating [smithi092.front.sepia.ceph.com]: reset os type and version on server
2024-03-28T23:27:23.058 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server
2024-03-28T23:27:23.079 INFO:teuthology.lock.ops:Start node 'smithi159.front.sepia.ceph.com' reimaging
2024-03-28T23:27:23.080 INFO:teuthology.lock.ops:Updating [smithi159.front.sepia.ceph.com]: reset os type and version on server
2024-03-28T23:27:23.080 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server
2024-03-28T23:27:23.222 DEBUG:teuthology.provision.fog.smithi159:Requesting image smithi_centos_9.stream (ID 10)
2024-03-28T23:27:23.226 DEBUG:teuthology.provision.fog.smithi092:Requesting image smithi_centos_9.stream (ID 10)
2024-03-28T23:27:23.267 INFO:teuthology.provision.fog.smithi092:Scheduling deploy of centos 9.stream
2024-03-28T23:27:23.274 INFO:teuthology.provision.fog.smithi159:Scheduling deploy of centos 9.stream
2024-03-28T23:27:23.512 INFO:teuthology.orchestra.console.smithi092:Power off
2024-03-28T23:27:23.513 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-28T23:27:23.529 INFO:teuthology.orchestra.console.smithi159:Power off
2024-03-28T23:27:23.529 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-28T23:27:23.538 DEBUG:teuthology.orchestra.console.smithi092:power off output: Chassis Power Control: Down/Off
2024-03-28T23:27:23.538 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-28T23:27:23.553 DEBUG:teuthology.orchestra.console.smithi159:power off output: Chassis Power Control: Down/Off
2024-03-28T23:27:23.553 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-28T23:27:23.563 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on
2024-03-28T23:27:23.574 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on
2024-03-28T23:27:27.569 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-28T23:27:27.576 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-28T23:27:27.679 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on
2024-03-28T23:27:27.685 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on
2024-03-28T23:27:31.684 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-28T23:27:31.691 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-28T23:27:31.793 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on
2024-03-28T23:27:31.799 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on
2024-03-28T23:27:35.798 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-28T23:27:35.805 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-28T23:27:35.907 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is off
2024-03-28T23:27:35.914 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off
2024-03-28T23:27:36.008 INFO:teuthology.orchestra.console.smithi092:Power off completed
2024-03-28T23:27:36.015 INFO:teuthology.orchestra.console.smithi159:Power off completed
2024-03-28T23:27:36.109 INFO:teuthology.orchestra.console.smithi092:Power on
2024-03-28T23:27:36.109 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-28T23:27:36.116 INFO:teuthology.orchestra.console.smithi159:Power on
2024-03-28T23:27:36.117 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-28T23:27:36.131 DEBUG:teuthology.orchestra.console.smithi092:power on output: Chassis Power Control: Up/On
2024-03-28T23:27:36.131 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-28T23:27:36.139 DEBUG:teuthology.orchestra.console.smithi159:power on output: Chassis Power Control: Up/On
2024-03-28T23:27:36.140 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-28T23:27:36.152 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is off
2024-03-28T23:27:36.161 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off
2024-03-28T23:27:40.155 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-28T23:27:40.163 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-28T23:27:40.265 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on
2024-03-28T23:27:40.272 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off
2024-03-28T23:27:40.366 INFO:teuthology.orchestra.console.smithi092:Power on completed
2024-03-28T23:27:40.467 INFO:teuthology.provision.fog.smithi092:Waiting for deploy to finish
2024-03-28T23:27:44.276 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-28T23:27:44.387 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on
2024-03-28T23:27:44.488 INFO:teuthology.orchestra.console.smithi159:Power on completed
2024-03-28T23:27:44.588 INFO:teuthology.provision.fog.smithi159:Waiting for deploy to finish
2024-03-28T23:30:11.386 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-28T23:30:30.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-28T23:30:33.538 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2024-03-28T23:30:39.539 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-28T23:30:42.590 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2024-03-28T23:30:48.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-28T23:30:51.646 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2024-03-28T23:30:57.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-28T23:31:00.706 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2024-03-28T23:31:06.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-28T23:31:09.762 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2024-03-28T23:31:15.764 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-28T23:31:18.814 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2024-03-28T23:31:24.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-28T23:31:27.870 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2024-03-28T23:31:33.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-28T23:31:36.930 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2024-03-28T23:31:42.067 ERROR:teuthology.orchestra.connection:Error authenticating with smithi092.front.sepia.ceph.com: AuthenticationException('Authentication timeout.')
2024-03-28T23:31:42.933 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-28T23:32:01.439 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)
2024-03-28T23:32:01.497 ERROR:teuthology.orchestra.connection:Error authenticating with smithi092.front.sepia.ceph.com: BadHostKeyException('smithi092.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QF1XTiR9/SchF9IhXH22Y6moZUPhyiEZkj0zX9hxmB4), PKey(alg=ED25519, bits=256, fp=SHA256:KbNe1Skfcs+enLKj/1/LNie9B1A7FfBvxciwHu0a4Lw)) (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect
    raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi092.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGTttKyG4J0sNWYCE0oLnnvvQUs7evZHxwYLfSDI2W8o5zdm7alJ1AOfneku2BHeqd08KghDhCVsZyEdt+QiKpQ=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAICtId6e6uguXOalpYiuQPFglwhHfKLWluuiTktQ8Xz60'
2024-03-28T23:32:08.536 ERROR:teuthology.orchestra.connection:Error authenticating with smithi092.front.sepia.ceph.com: BadHostKeyException('smithi092.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QF1XTiR9/SchF9IhXH22Y6moZUPhyiEZkj0zX9hxmB4), PKey(alg=ED25519, bits=256, fp=SHA256:KbNe1Skfcs+enLKj/1/LNie9B1A7FfBvxciwHu0a4Lw)) (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect
    raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi092.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGTttKyG4J0sNWYCE0oLnnvvQUs7evZHxwYLfSDI2W8o5zdm7alJ1AOfneku2BHeqd08KghDhCVsZyEdt+QiKpQ=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAICtId6e6uguXOalpYiuQPFglwhHfKLWluuiTktQ8Xz60'
2024-03-28T23:32:18.556 ERROR:teuthology.orchestra.connection:Error authenticating with smithi092.front.sepia.ceph.com: BadHostKeyException('smithi092.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QF1XTiR9/SchF9IhXH22Y6moZUPhyiEZkj0zX9hxmB4), PKey(alg=ED25519, bits=256, fp=SHA256:KbNe1Skfcs+enLKj/1/LNie9B1A7FfBvxciwHu0a4Lw)) (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect
    raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi092.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGTttKyG4J0sNWYCE0oLnnvvQUs7evZHxwYLfSDI2W8o5zdm7alJ1AOfneku2BHeqd08KghDhCVsZyEdt+QiKpQ=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAICtId6e6uguXOalpYiuQPFglwhHfKLWluuiTktQ8Xz60'
2024-03-28T23:32:31.575 ERROR:teuthology.orchestra.connection:Error authenticating with smithi092.front.sepia.ceph.com: BadHostKeyException('smithi092.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QF1XTiR9/SchF9IhXH22Y6moZUPhyiEZkj0zX9hxmB4), PKey(alg=ED25519, bits=256, fp=SHA256:KbNe1Skfcs+enLKj/1/LNie9B1A7FfBvxciwHu0a4Lw)) (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect
    raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi092.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGTttKyG4J0sNWYCE0oLnnvvQUs7evZHxwYLfSDI2W8o5zdm7alJ1AOfneku2BHeqd08KghDhCVsZyEdt+QiKpQ=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAICtId6e6uguXOalpYiuQPFglwhHfKLWluuiTktQ8Xz60'
2024-03-28T23:32:42.936 WARNING:teuthology.provision.fog.smithi159:timed out
2024-03-28T23:32:47.593 ERROR:teuthology.orchestra.connection:Error authenticating with smithi092.front.sepia.ceph.com: BadHostKeyException('smithi092.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QF1XTiR9/SchF9IhXH22Y6moZUPhyiEZkj0zX9hxmB4), PKey(alg=ED25519, bits=256, fp=SHA256:KbNe1Skfcs+enLKj/1/LNie9B1A7FfBvxciwHu0a4Lw)) (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect
    raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi092.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGTttKyG4J0sNWYCE0oLnnvvQUs7evZHxwYLfSDI2W8o5zdm7alJ1AOfneku2BHeqd08KghDhCVsZyEdt+QiKpQ=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAICtId6e6uguXOalpYiuQPFglwhHfKLWluuiTktQ8Xz60'
2024-03-28T23:32:48.937 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-28T23:32:49.307 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-28T23:32:49.308 DEBUG:teuthology.orchestra.run.smithi159:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-28T23:32:49.625 INFO:teuthology.provision.fog.smithi159:Node is ready
2024-03-28T23:32:49.625 DEBUG:teuthology.orchestra.run.smithi159:> hostname
2024-03-28T23:32:49.639 INFO:teuthology.orchestra.run.smithi159.stdout:smithi159.front.sepia.ceph.com
2024-03-28T23:32:49.640 DEBUG:teuthology.orchestra.run.smithi159:> grep smithi159.front.sepia.ceph.com /etc/hosts
2024-03-28T23:32:49.696 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.15.159 smithi159.front.sepia.ceph.com smithi159
2024-03-28T23:32:49.696 DEBUG:teuthology.orchestra.run.smithi159:> sudo hostname smithi159
2024-03-28T23:32:49.804 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i -e 's/smithi159.front.sepia.ceph.com/smithi159/g' /etc/hosts
2024-03-28T23:32:49.832 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i -e 's/172.21.15.159/172.21.15.159/g' /etc/hosts
2024-03-28T23:32:49.900 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i -e 's/smithi159.front.sepia.ceph.com/smithi159/g' /etc/hostname
2024-03-28T23:32:49.968 DEBUG:teuthology.orchestra.run.smithi159:> sudo hostname smithi159
2024-03-28T23:32:50.035 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release
2024-03-28T23:32:50.090 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="CentOS Stream"
2024-03-28T23:32:50.090 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="9"
2024-03-28T23:32:50.090 INFO:teuthology.orchestra.run.smithi159.stdout:ID="centos"
2024-03-28T23:32:50.091 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="rhel fedora"
2024-03-28T23:32:50.091 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="9"
2024-03-28T23:32:50.091 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el9"
2024-03-28T23:32:50.091 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-28T23:32:50.091 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31"
2024-03-28T23:32:50.091 INFO:teuthology.orchestra.run.smithi159.stdout:LOGO="fedora-logo-icon"
2024-03-28T23:32:50.091 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-28T23:32:50.091 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://centos.org/"
2024-03-28T23:32:50.091 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-28T23:32:50.091 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-28T23:32:50.091 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-28T23:32:50.092 INFO:teuthology.provision.fog.smithi159:Deploy complete!
2024-03-28T23:32:50.094 DEBUG:teuthology.parallel:result is None
2024-03-28T23:33:06.611 ERROR:teuthology.orchestra.connection:Error authenticating with smithi092.front.sepia.ceph.com: BadHostKeyException('smithi092.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QF1XTiR9/SchF9IhXH22Y6moZUPhyiEZkj0zX9hxmB4), PKey(alg=ED25519, bits=256, fp=SHA256:KbNe1Skfcs+enLKj/1/LNie9B1A7FfBvxciwHu0a4Lw)) (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect
    raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi092.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGTttKyG4J0sNWYCE0oLnnvvQUs7evZHxwYLfSDI2W8o5zdm7alJ1AOfneku2BHeqd08KghDhCVsZyEdt+QiKpQ=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAICtId6e6uguXOalpYiuQPFglwhHfKLWluuiTktQ8Xz60'
2024-03-28T23:33:28.630 ERROR:teuthology.orchestra.connection:Error authenticating with smithi092.front.sepia.ceph.com: BadHostKeyException('smithi092.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QF1XTiR9/SchF9IhXH22Y6moZUPhyiEZkj0zX9hxmB4), PKey(alg=ED25519, bits=256, fp=SHA256:KbNe1Skfcs+enLKj/1/LNie9B1A7FfBvxciwHu0a4Lw)) (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect
    raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi092.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGTttKyG4J0sNWYCE0oLnnvvQUs7evZHxwYLfSDI2W8o5zdm7alJ1AOfneku2BHeqd08KghDhCVsZyEdt+QiKpQ=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAICtId6e6uguXOalpYiuQPFglwhHfKLWluuiTktQ8Xz60'
2024-03-28T23:33:53.651 ERROR:teuthology.orchestra.connection:Error authenticating with smithi092.front.sepia.ceph.com: BadHostKeyException('smithi092.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QF1XTiR9/SchF9IhXH22Y6moZUPhyiEZkj0zX9hxmB4), PKey(alg=ED25519, bits=256, fp=SHA256:KbNe1Skfcs+enLKj/1/LNie9B1A7FfBvxciwHu0a4Lw)) (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect
    raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi092.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGTttKyG4J0sNWYCE0oLnnvvQUs7evZHxwYLfSDI2W8o5zdm7alJ1AOfneku2BHeqd08KghDhCVsZyEdt+QiKpQ=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAICtId6e6uguXOalpYiuQPFglwhHfKLWluuiTktQ8Xz60'
2024-03-28T23:34:21.712 ERROR:teuthology.orchestra.connection:Error authenticating with smithi092.front.sepia.ceph.com: BadHostKeyException('smithi092.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:QF1XTiR9/SchF9IhXH22Y6moZUPhyiEZkj0zX9hxmB4), PKey(alg=ED25519, bits=256, fp=SHA256:KbNe1Skfcs+enLKj/1/LNie9B1A7FfBvxciwHu0a4Lw)) (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect
    raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi092.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGTttKyG4J0sNWYCE0oLnnvvQUs7evZHxwYLfSDI2W8o5zdm7alJ1AOfneku2BHeqd08KghDhCVsZyEdt+QiKpQ=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAICtId6e6uguXOalpYiuQPFglwhHfKLWluuiTktQ8Xz60'
2024-03-28T23:34:21.713 WARNING:teuthology.provision.fog.smithi092:'connect to smithi092.front.sepia.ceph.com' reached maximum tries (11) after waiting for 145 seconds
2024-03-28T23:34:27.714 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-28T23:34:27.908 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-28T23:34:27.908 DEBUG:teuthology.orchestra.run.smithi092:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-28T23:34:28.260 INFO:teuthology.provision.fog.smithi092:Node is ready
2024-03-28T23:34:28.261 DEBUG:teuthology.orchestra.run.smithi092:> hostname
2024-03-28T23:34:28.275 INFO:teuthology.orchestra.run.smithi092.stdout:smithi092.front.sepia.ceph.com
2024-03-28T23:34:28.275 DEBUG:teuthology.orchestra.run.smithi092:> grep smithi092.front.sepia.ceph.com /etc/hosts
2024-03-28T23:34:28.331 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.92 smithi092.front.sepia.ceph.com smithi092
2024-03-28T23:34:28.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo hostname smithi092
2024-03-28T23:34:28.435 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i -e 's/smithi092.front.sepia.ceph.com/smithi092/g' /etc/hosts
2024-03-28T23:34:28.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i -e 's/172.21.15.92/172.21.15.92/g' /etc/hosts
2024-03-28T23:34:28.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i -e 's/smithi092.front.sepia.ceph.com/smithi092/g' /etc/hostname
2024-03-28T23:34:28.598 DEBUG:teuthology.orchestra.run.smithi092:> sudo hostname smithi092
2024-03-28T23:34:28.664 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release
2024-03-28T23:34:28.719 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="CentOS Stream"
2024-03-28T23:34:28.719 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="9"
2024-03-28T23:34:28.719 INFO:teuthology.orchestra.run.smithi092.stdout:ID="centos"
2024-03-28T23:34:28.719 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="rhel fedora"
2024-03-28T23:34:28.720 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="9"
2024-03-28T23:34:28.720 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el9"
2024-03-28T23:34:28.720 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-28T23:34:28.720 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31"
2024-03-28T23:34:28.720 INFO:teuthology.orchestra.run.smithi092.stdout:LOGO="fedora-logo-icon"
2024-03-28T23:34:28.720 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-28T23:34:28.720 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://centos.org/"
2024-03-28T23:34:28.720 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-28T23:34:28.720 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-28T23:34:28.720 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-28T23:34:28.721 INFO:teuthology.provision.fog.smithi092:Deploy complete!
2024-03-28T23:34:28.721 DEBUG:teuthology.parallel:result is None
2024-03-28T23:34:28.915 INFO:teuthology.lock.ops:Checking smithi092.front.sepia.ceph.com
2024-03-28T23:34:28.916 INFO:teuthology.lock.ops:New key found. Updating...
2024-03-28T23:34:28.941 INFO:teuthology.lock.ops:Checking smithi159.front.sepia.ceph.com
2024-03-28T23:34:28.942 INFO:teuthology.lock.ops:New key found. Updating...
2024-03-28T23:34:28.959 INFO:teuthology.lock.ops:Updating [smithi092.front.sepia.ceph.com]: set os type and version on server
2024-03-28T23:34:28.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com'
2024-03-28T23:34:28.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-28T23:34:29.046 DEBUG:teuthology.orchestra.run.smithi092:> true
2024-03-28T23:34:29.069 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com'
2024-03-28T23:34:29.069 DEBUG:teuthology.orchestra.run.smithi092:> uname -m
2024-03-28T23:34:29.122 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64
2024-03-28T23:34:29.123 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release
2024-03-28T23:34:29.177 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="CentOS Stream"
2024-03-28T23:34:29.177 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="9"
2024-03-28T23:34:29.177 INFO:teuthology.orchestra.run.smithi092.stdout:ID="centos"
2024-03-28T23:34:29.177 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="rhel fedora"
2024-03-28T23:34:29.177 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="9"
2024-03-28T23:34:29.178 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el9"
2024-03-28T23:34:29.178 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-28T23:34:29.178 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31"
2024-03-28T23:34:29.178 INFO:teuthology.orchestra.run.smithi092.stdout:LOGO="fedora-logo-icon"
2024-03-28T23:34:29.178 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-28T23:34:29.178 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://centos.org/"
2024-03-28T23:34:29.178 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-28T23:34:29.178 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-28T23:34:29.178 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-28T23:34:29.179 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server
2024-03-28T23:34:29.200 INFO:teuthology.lock.ops:Updating [smithi159.front.sepia.ceph.com]: set os type and version on server
2024-03-28T23:34:29.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2024-03-28T23:34:29.200 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-28T23:34:29.287 DEBUG:teuthology.orchestra.run.smithi159:> true
2024-03-28T23:34:29.456 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com'
2024-03-28T23:34:29.456 DEBUG:teuthology.orchestra.run.smithi159:> uname -m
2024-03-28T23:34:29.511 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64
2024-03-28T23:34:29.511 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release
2024-03-28T23:34:29.566 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="CentOS Stream"
2024-03-28T23:34:29.566 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="9"
2024-03-28T23:34:29.566 INFO:teuthology.orchestra.run.smithi159.stdout:ID="centos"
2024-03-28T23:34:29.566 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="rhel fedora"
2024-03-28T23:34:29.566 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="9"
2024-03-28T23:34:29.566 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el9"
2024-03-28T23:34:29.566 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-28T23:34:29.566 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31"
2024-03-28T23:34:29.566 INFO:teuthology.orchestra.run.smithi159.stdout:LOGO="fedora-logo-icon"
2024-03-28T23:34:29.566 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-28T23:34:29.566 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://centos.org/"
2024-03-28T23:34:29.567 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-28T23:34:29.567 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-28T23:34:29.567 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-28T23:34:29.567 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server
2024-03-28T23:34:29.591 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-28T23:34:29.676 INFO:teuthology.dispatcher.supervisor:Running job 7627215
2024-03-28T23:34:29.676 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627215 --name teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627215/orig.config.yaml
2024-03-28T23:34:29.679 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627215
2024-03-28T23:34:29.679 INFO:teuthology.dispatcher.supervisor:Job PID: 3103782
2024-03-28T23:34:29.680 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2024-03-28T23:36:29.711 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-28T23:38:29.879 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-28T23:40:30.055 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-28T23:42:30.227 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-28T23:44:30.399 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-28T23:46:30.573 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-28T23:48:30.743 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-28T23:50:31.095 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-28T23:52:31.262 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-28T23:54:31.423 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-28T23:56:31.588 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-28T23:58:31.762 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-29T00:00:31.944 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-29T00:02:32.115 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-29T00:04:32.283 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-29T00:06:32.447 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-29T00:08:32.619 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-29T00:10:32.795 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-29T00:12:32.964 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-29T00:14:33.143 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-29T00:16:33.307 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-29T00:18:33.475 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-29T00:18:33.545 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1
2024-03-29T00:18:33.724 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi092 but it was locked by another job: /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627271
2024-03-29T00:18:33.724 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi159 but it was locked by another job: /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627270