2024-02-06T19:50:41.104 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-06T19:50:41.104 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-06T19:50:41.309 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcc3b4fac10>, signals=[15]) 2024-02-06T19:50:41.309 INFO:teuthology.lock.ops:Start node 'smithi078.front.sepia.ceph.com' reimaging 2024-02-06T19:50:41.309 INFO:teuthology.lock.ops:Updating [smithi078.front.sepia.ceph.com]: reset os type and version on server 2024-02-06T19:50:41.309 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-02-06T19:50:41.334 INFO:teuthology.lock.ops:Start node 'smithi131.front.sepia.ceph.com' reimaging 2024-02-06T19:50:41.334 INFO:teuthology.lock.ops:Updating [smithi131.front.sepia.ceph.com]: reset os type and version on server 2024-02-06T19:50:41.334 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2024-02-06T19:50:41.358 INFO:teuthology.lock.ops:Start node 'smithi176.front.sepia.ceph.com' reimaging 2024-02-06T19:50:41.359 INFO:teuthology.lock.ops:Updating [smithi176.front.sepia.ceph.com]: reset os type and version on server 2024-02-06T19:50:41.359 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-02-06T19:50:41.454 DEBUG:teuthology.provision.fog.smithi078:Requesting image smithi_centos_9.stream (ID 10) 2024-02-06T19:50:41.474 DEBUG:teuthology.provision.fog.smithi131:Requesting image smithi_centos_9.stream (ID 10) 2024-02-06T19:50:41.494 DEBUG:teuthology.provision.fog.smithi176:Requesting image smithi_centos_9.stream (ID 10) 2024-02-06T19:50:41.498 INFO:teuthology.provision.fog.smithi078:Scheduling deploy of centos 9.stream 2024-02-06T19:50:41.510 INFO:teuthology.provision.fog.smithi131:Scheduling deploy of centos 9.stream 2024-02-06T19:50:41.540 INFO:teuthology.provision.fog.smithi176:Scheduling deploy of centos 9.stream 2024-02-06T19:50:41.719 INFO:teuthology.orchestra.console.smithi078:Power off 2024-02-06T19:50:41.720 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-06T19:50:41.744 INFO:teuthology.orchestra.console.smithi131:Power off 2024-02-06T19:50:41.745 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-06T19:50:41.762 DEBUG:teuthology.orchestra.console.smithi078:power off output: Chassis Power Control: Down/Off 2024-02-06T19:50:41.762 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:41.771 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off 2024-02-06T19:50:41.771 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:41.786 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-02-06T19:50:41.795 INFO:teuthology.orchestra.console.smithi176:Power off 2024-02-06T19:50:41.795 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-06T19:50:41.803 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-02-06T19:50:41.837 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2024-02-06T19:50:41.837 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:41.860 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-02-06T19:50:45.788 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:45.804 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:45.861 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:45.898 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-02-06T19:50:45.913 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-02-06T19:50:45.970 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-02-06T19:50:49.903 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:49.914 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:49.971 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:50.013 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off 2024-02-06T19:50:50.023 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2024-02-06T19:50:50.079 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-02-06T19:50:50.114 INFO:teuthology.orchestra.console.smithi078:Power off completed 2024-02-06T19:50:50.124 INFO:teuthology.orchestra.console.smithi131:Power off completed 2024-02-06T19:50:50.215 INFO:teuthology.orchestra.console.smithi078:Power on 2024-02-06T19:50:50.215 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-02-06T19:50:50.226 INFO:teuthology.orchestra.console.smithi131:Power on 2024-02-06T19:50:50.226 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-02-06T19:50:50.237 DEBUG:teuthology.orchestra.console.smithi078:power on output: Chassis Power Control: Up/On 2024-02-06T19:50:50.238 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:50.249 DEBUG:teuthology.orchestra.console.smithi131:power on output: Chassis Power Control: Up/On 2024-02-06T19:50:50.249 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:50.260 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off 2024-02-06T19:50:50.271 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2024-02-06T19:50:54.084 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:54.194 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-02-06T19:50:54.261 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:54.273 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:54.295 INFO:teuthology.orchestra.console.smithi176:Power off completed 2024-02-06T19:50:54.372 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-02-06T19:50:54.382 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-02-06T19:50:54.396 INFO:teuthology.orchestra.console.smithi176:Power on 2024-02-06T19:50:54.396 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-02-06T19:50:54.420 DEBUG:teuthology.orchestra.console.smithi176:power on output: Chassis Power Control: Up/On 2024-02-06T19:50:54.420 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:54.443 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-02-06T19:50:54.472 INFO:teuthology.orchestra.console.smithi078:Power on completed 2024-02-06T19:50:54.483 INFO:teuthology.orchestra.console.smithi131:Power on completed 2024-02-06T19:50:54.573 INFO:teuthology.provision.fog.smithi078:Waiting for deploy to finish 2024-02-06T19:50:54.584 INFO:teuthology.provision.fog.smithi131:Waiting for deploy to finish 2024-02-06T19:50:58.448 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T19:50:58.557 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-02-06T19:50:58.657 INFO:teuthology.orchestra.console.smithi176:Power on completed 2024-02-06T19:50:58.758 INFO:teuthology.provision.fog.smithi176:Waiting for deploy to finish 2024-02-06T19:53:25.340 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T19:53:29.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T19:53:29.541 ERROR:teuthology.orchestra.connection:Error authenticating with smithi176.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-02-06T19:53:40.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T19:53:43.452 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131 2024-02-06T19:53:49.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T19:53:52.512 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131 2024-02-06T19:53:58.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T19:54:01.564 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131 2024-02-06T19:54:07.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T19:54:10.623 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131 2024-02-06T19:54:16.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T19:54:19.676 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131 2024-02-06T19:54:25.401 ERROR:teuthology.orchestra.connection:Error authenticating with smithi078.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 457, in connect server_key = t.get_remote_server_key() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 918, in get_remote_server_key raise SSHException("No existing session") paramiko.ssh_exception.SSHException: No existing session 2024-02-06T19:54:25.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T19:54:33.544 WARNING:teuthology.provision.fog.smithi176:timed out 2024-02-06T19:54:39.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T19:55:13.340 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-02-06T19:55:25.679 WARNING:teuthology.provision.fog.smithi131:timed out 2024-02-06T19:55:29.405 WARNING:teuthology.provision.fog.smithi078:timed out 2024-02-06T19:55:31.681 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T19:55:31.938 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-02-06T19:55:31.939 DEBUG:teuthology.orchestra.run.smithi131:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-02-06T19:55:32.253 INFO:teuthology.provision.fog.smithi131:Node is ready 2024-02-06T19:55:32.254 DEBUG:teuthology.orchestra.run.smithi131:> hostname 2024-02-06T19:55:32.272 INFO:teuthology.orchestra.run.smithi131.stdout:smithi131.front.sepia.ceph.com 2024-02-06T19:55:32.272 DEBUG:teuthology.orchestra.run.smithi131:> grep smithi131.front.sepia.ceph.com /etc/hosts 2024-02-06T19:55:32.328 INFO:teuthology.orchestra.run.smithi131.stdout:172.21.15.131 smithi131.front.sepia.ceph.com smithi131 2024-02-06T19:55:32.328 DEBUG:teuthology.orchestra.run.smithi131:> sudo hostname smithi131 2024-02-06T19:55:32.449 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i -e 's/smithi131.front.sepia.ceph.com/smithi131/g' /etc/hosts 2024-02-06T19:55:32.480 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i -e 's/172.21.15.131/172.21.15.131/g' /etc/hosts 2024-02-06T19:55:32.550 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i -e 's/smithi131.front.sepia.ceph.com/smithi131/g' /etc/hostname 2024-02-06T19:55:32.618 DEBUG:teuthology.orchestra.run.smithi131:> sudo hostname smithi131 2024-02-06T19:55:32.688 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2024-02-06T19:55:32.743 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="CentOS Stream" 2024-02-06T19:55:32.743 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="9" 2024-02-06T19:55:32.743 INFO:teuthology.orchestra.run.smithi131.stdout:ID="centos" 2024-02-06T19:55:32.744 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="rhel fedora" 2024-02-06T19:55:32.744 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="9" 2024-02-06T19:55:32.744 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el9" 2024-02-06T19:55:32.744 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-06T19:55:32.744 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2024-02-06T19:55:32.744 INFO:teuthology.orchestra.run.smithi131.stdout:LOGO="fedora-logo-icon" 2024-02-06T19:55:32.744 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-06T19:55:32.744 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://centos.org/" 2024-02-06T19:55:32.744 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-06T19:55:32.744 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-06T19:55:32.744 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-06T19:55:32.745 INFO:teuthology.provision.fog.smithi131:Deploy complete! 2024-02-06T19:55:32.747 DEBUG:teuthology.parallel:result is None 2024-02-06T19:55:35.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T19:55:35.758 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-02-06T19:55:35.758 DEBUG:teuthology.orchestra.run.smithi078:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-02-06T19:55:36.074 INFO:teuthology.provision.fog.smithi078:Node is ready 2024-02-06T19:55:36.074 DEBUG:teuthology.orchestra.run.smithi078:> hostname 2024-02-06T19:55:36.088 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078.front.sepia.ceph.com 2024-02-06T19:55:36.088 DEBUG:teuthology.orchestra.run.smithi078:> grep smithi078.front.sepia.ceph.com /etc/hosts 2024-02-06T19:55:36.143 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78 smithi078.front.sepia.ceph.com smithi078 2024-02-06T19:55:36.143 DEBUG:teuthology.orchestra.run.smithi078:> sudo hostname smithi078 2024-02-06T19:55:36.252 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i -e 's/smithi078.front.sepia.ceph.com/smithi078/g' /etc/hosts 2024-02-06T19:55:36.279 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i -e 's/172.21.15.78/172.21.15.78/g' /etc/hosts 2024-02-06T19:55:36.346 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i -e 's/smithi078.front.sepia.ceph.com/smithi078/g' /etc/hostname 2024-02-06T19:55:36.414 DEBUG:teuthology.orchestra.run.smithi078:> sudo hostname smithi078 2024-02-06T19:55:36.480 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-02-06T19:55:36.535 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream" 2024-02-06T19:55:36.535 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="9" 2024-02-06T19:55:36.535 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2024-02-06T19:55:36.535 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2024-02-06T19:55:36.535 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="9" 2024-02-06T19:55:36.535 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el9" 2024-02-06T19:55:36.535 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-06T19:55:36.535 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2024-02-06T19:55:36.535 INFO:teuthology.orchestra.run.smithi078.stdout:LOGO="fedora-logo-icon" 2024-02-06T19:55:36.536 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-06T19:55:36.536 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2024-02-06T19:55:36.536 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-06T19:55:36.536 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-06T19:55:36.536 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-06T19:55:36.536 INFO:teuthology.provision.fog.smithi078:Deploy complete! 2024-02-06T19:55:36.537 DEBUG:teuthology.parallel:result is None 2024-02-06T19:55:39.549 WARNING:teuthology.provision.fog.smithi176:timed out 2024-02-06T19:55:45.551 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T19:55:45.758 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-02-06T19:55:45.758 DEBUG:teuthology.orchestra.run.smithi176:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-02-06T19:55:46.091 INFO:teuthology.provision.fog.smithi176:Node is ready 2024-02-06T19:55:46.092 DEBUG:teuthology.orchestra.run.smithi176:> hostname 2024-02-06T19:55:46.107 INFO:teuthology.orchestra.run.smithi176.stdout:smithi176.front.sepia.ceph.com 2024-02-06T19:55:46.108 DEBUG:teuthology.orchestra.run.smithi176:> grep smithi176.front.sepia.ceph.com /etc/hosts 2024-02-06T19:55:46.164 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.15.176 smithi176.front.sepia.ceph.com smithi176 2024-02-06T19:55:46.164 DEBUG:teuthology.orchestra.run.smithi176:> sudo hostname smithi176 2024-02-06T19:55:46.263 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i -e 's/smithi176.front.sepia.ceph.com/smithi176/g' /etc/hosts 2024-02-06T19:55:46.294 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i -e 's/172.21.15.176/172.21.15.176/g' /etc/hosts 2024-02-06T19:55:46.363 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i -e 's/smithi176.front.sepia.ceph.com/smithi176/g' /etc/hostname 2024-02-06T19:55:46.435 DEBUG:teuthology.orchestra.run.smithi176:> sudo hostname smithi176 2024-02-06T19:55:46.504 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-02-06T19:55:46.563 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2024-02-06T19:55:46.563 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="9" 2024-02-06T19:55:46.563 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2024-02-06T19:55:46.563 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2024-02-06T19:55:46.563 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="9" 2024-02-06T19:55:46.563 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el9" 2024-02-06T19:55:46.563 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-06T19:55:46.563 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-02-06T19:55:46.564 INFO:teuthology.orchestra.run.smithi176.stdout:LOGO="fedora-logo-icon" 2024-02-06T19:55:46.564 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-06T19:55:46.564 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2024-02-06T19:55:46.564 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-06T19:55:46.564 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-06T19:55:46.564 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-06T19:55:46.564 INFO:teuthology.provision.fog.smithi176:Deploy complete! 2024-02-06T19:55:46.565 DEBUG:teuthology.parallel:result is None 2024-02-06T19:55:46.779 INFO:teuthology.lock.ops:Checking smithi078.front.sepia.ceph.com 2024-02-06T19:55:46.779 INFO:teuthology.lock.ops:New key found. Updating... 2024-02-06T19:55:46.820 INFO:teuthology.lock.ops:Checking smithi131.front.sepia.ceph.com 2024-02-06T19:55:46.820 INFO:teuthology.lock.ops:New key found. Updating... 2024-02-06T19:55:46.841 INFO:teuthology.lock.ops:Checking smithi176.front.sepia.ceph.com 2024-02-06T19:55:46.842 INFO:teuthology.lock.ops:New key found. Updating... 2024-02-06T19:55:46.867 INFO:teuthology.lock.ops:Updating [smithi078.front.sepia.ceph.com]: set os type and version on server 2024-02-06T19:55:46.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-02-06T19:55:46.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T19:55:46.963 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-02-06T19:55:47.131 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-02-06T19:55:47.131 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2024-02-06T19:55:47.186 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2024-02-06T19:55:47.186 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-02-06T19:55:47.241 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream" 2024-02-06T19:55:47.241 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="9" 2024-02-06T19:55:47.241 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2024-02-06T19:55:47.242 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2024-02-06T19:55:47.242 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="9" 2024-02-06T19:55:47.242 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el9" 2024-02-06T19:55:47.242 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-06T19:55:47.242 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2024-02-06T19:55:47.242 INFO:teuthology.orchestra.run.smithi078.stdout:LOGO="fedora-logo-icon" 2024-02-06T19:55:47.242 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-06T19:55:47.242 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2024-02-06T19:55:47.242 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-06T19:55:47.242 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-06T19:55:47.242 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-06T19:55:47.243 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-02-06T19:55:47.267 INFO:teuthology.lock.ops:Updating [smithi131.front.sepia.ceph.com]: set os type and version on server 2024-02-06T19:55:47.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-02-06T19:55:47.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T19:55:47.367 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-02-06T19:55:47.534 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-02-06T19:55:47.534 DEBUG:teuthology.orchestra.run.smithi131:> uname -m 2024-02-06T19:55:47.588 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2024-02-06T19:55:47.588 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2024-02-06T19:55:47.642 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="CentOS Stream" 2024-02-06T19:55:47.642 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="9" 2024-02-06T19:55:47.642 INFO:teuthology.orchestra.run.smithi131.stdout:ID="centos" 2024-02-06T19:55:47.642 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="rhel fedora" 2024-02-06T19:55:47.643 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="9" 2024-02-06T19:55:47.643 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el9" 2024-02-06T19:55:47.643 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-06T19:55:47.643 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2024-02-06T19:55:47.643 INFO:teuthology.orchestra.run.smithi131.stdout:LOGO="fedora-logo-icon" 2024-02-06T19:55:47.643 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-06T19:55:47.643 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://centos.org/" 2024-02-06T19:55:47.643 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-06T19:55:47.643 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-06T19:55:47.643 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-06T19:55:47.644 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2024-02-06T19:55:47.667 INFO:teuthology.lock.ops:Updating [smithi176.front.sepia.ceph.com]: set os type and version on server 2024-02-06T19:55:47.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-02-06T19:55:47.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T19:55:47.760 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-02-06T19:55:47.828 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-02-06T19:55:47.828 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-02-06T19:55:47.889 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-02-06T19:55:47.889 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-02-06T19:55:47.948 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2024-02-06T19:55:47.948 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="9" 2024-02-06T19:55:47.948 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2024-02-06T19:55:47.949 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2024-02-06T19:55:47.949 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="9" 2024-02-06T19:55:47.949 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el9" 2024-02-06T19:55:47.949 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-06T19:55:47.949 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-02-06T19:55:47.949 INFO:teuthology.orchestra.run.smithi176.stdout:LOGO="fedora-logo-icon" 2024-02-06T19:55:47.949 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-06T19:55:47.949 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2024-02-06T19:55:47.949 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-06T19:55:47.950 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-06T19:55:47.950 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-06T19:55:47.950 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-02-06T19:55:47.975 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-06T19:55:48.080 INFO:teuthology.dispatcher.supervisor:Running job 7548264 2024-02-06T19:55:48.081 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-02-06_01:32:07-rados-wip-yuri6-testing-2024-02-05-1523-distro-default-smithi/7548264 --name yuriw-2024-02-06_01:32:07-rados-wip-yuri6-testing-2024-02-05-1523-distro-default-smithi --description rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} -- /home/teuthworker/archive/yuriw-2024-02-06_01:32:07-rados-wip-yuri6-testing-2024-02-05-1523-distro-default-smithi/7548264/orig.config.yaml 2024-02-06T19:55:48.085 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2024-02-06_01:32:07-rados-wip-yuri6-testing-2024-02-05-1523-distro-default-smithi/7548264 2024-02-06T19:55:48.085 INFO:teuthology.dispatcher.supervisor:Job PID: 3342772 2024-02-06T19:55:48.085 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2024-02-06T19:57:48.087 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-06T19:59:48.170 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-06T20:01:48.248 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-06T20:03:48.324 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-06T20:05:48.398 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-06T20:07:48.463 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-06T20:09:48.539 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-06T20:09:48.613 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1