2024-03-08T05:35:49.116 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-08T05:35:49.116 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-08T05:35:49.251 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2d2f853c10>, signals=[15]) 2024-03-08T05:35:49.252 INFO:teuthology.lock.ops:Start node 'smithi053.front.sepia.ceph.com' reimaging 2024-03-08T05:35:49.252 INFO:teuthology.lock.ops:Updating [smithi053.front.sepia.ceph.com]: reset os type and version on server 2024-03-08T05:35:49.252 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-03-08T05:35:49.274 INFO:teuthology.lock.ops:Start node 'smithi098.front.sepia.ceph.com' reimaging 2024-03-08T05:35:49.275 INFO:teuthology.lock.ops:Updating [smithi098.front.sepia.ceph.com]: reset os type and version on server 2024-03-08T05:35:49.275 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-03-08T05:35:49.383 DEBUG:teuthology.provision.fog.smithi053:Requesting image smithi_centos_9.stream (ID 10) 2024-03-08T05:35:49.402 DEBUG:teuthology.provision.fog.smithi098:Requesting image smithi_centos_9.stream (ID 10) 2024-03-08T05:35:49.423 INFO:teuthology.provision.fog.smithi053:Scheduling deploy of centos 9.stream 2024-03-08T05:35:49.447 INFO:teuthology.provision.fog.smithi098:Scheduling deploy of centos 9.stream 2024-03-08T05:35:49.727 INFO:teuthology.orchestra.console.smithi098:Power off 2024-03-08T05:35:49.727 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-08T05:35:49.743 INFO:teuthology.orchestra.console.smithi053:Power off 2024-03-08T05:35:49.743 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-08T05:35:49.750 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-03-08T05:35:49.750 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T05:35:49.763 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-03-08T05:35:49.763 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T05:35:49.772 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-03-08T05:35:49.784 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-08T05:35:49.873 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-03-08T05:35:49.885 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-03-08T05:35:49.974 INFO:teuthology.orchestra.console.smithi098:Power on 2024-03-08T05:35:49.975 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-08T05:35:49.986 INFO:teuthology.orchestra.console.smithi053:Power on 2024-03-08T05:35:49.986 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-08T05:35:49.995 DEBUG:teuthology.orchestra.console.smithi098:power on output: Chassis Power Control: Up/On 2024-03-08T05:35:49.996 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T05:35:50.006 DEBUG:teuthology.orchestra.console.smithi053:power on output: Chassis Power Control: Up/On 2024-03-08T05:35:50.006 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T05:35:50.018 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-03-08T05:35:50.035 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-08T05:35:54.022 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T05:35:54.036 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T05:35:54.133 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-03-08T05:35:54.147 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-08T05:35:54.249 INFO:teuthology.orchestra.console.smithi053:Power on completed 2024-03-08T05:35:54.350 INFO:teuthology.provision.fog.smithi053:Waiting for deploy to finish 2024-03-08T05:35:58.137 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T05:35:58.245 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-08T05:35:58.346 INFO:teuthology.orchestra.console.smithi098:Power on completed 2024-03-08T05:35:58.447 INFO:teuthology.provision.fog.smithi098:Waiting for deploy to finish 2024-03-08T05:44:46.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T05:45:05.055 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-08T05:45:11.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T05:45:14.142 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-08T05:45:20.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T05:45:23.231 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-08T05:45:29.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T05:45:32.319 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-08T05:45:38.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T05:45:56.766 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-08T05:46:02.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T05:46:05.822 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-08T05:46:11.829 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T05:46:11.959 ERROR:teuthology.orchestra.connection:Error authenticating with smithi053.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-03-08T05:46:16.045 ERROR:teuthology.orchestra.connection:Error authenticating with smithi053.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-03-08T05:46:23.131 ERROR:teuthology.orchestra.connection:Error authenticating with smithi053.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-03-08T05:46:33.257 ERROR:teuthology.orchestra.connection:Error authenticating with smithi053.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-03-08T05:46:46.344 ERROR:teuthology.orchestra.connection:Error authenticating with smithi053.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-03-08T05:47:34.174 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-08T05:47:40.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T05:47:43.230 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-08T05:47:49.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T05:48:49.234 WARNING:teuthology.provision.fog.smithi053:timed out 2024-03-08T05:48:55.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T05:48:55.500 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-08T05:48:55.501 DEBUG:teuthology.orchestra.run.smithi053:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-08T05:48:55.839 INFO:teuthology.provision.fog.smithi053:Node is ready 2024-03-08T05:48:55.840 DEBUG:teuthology.orchestra.run.smithi053:> hostname 2024-03-08T05:48:55.854 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053.front.sepia.ceph.com 2024-03-08T05:48:55.855 DEBUG:teuthology.orchestra.run.smithi053:> grep smithi053.front.sepia.ceph.com /etc/hosts 2024-03-08T05:48:55.910 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53 smithi053.front.sepia.ceph.com smithi053 2024-03-08T05:48:55.910 DEBUG:teuthology.orchestra.run.smithi053:> sudo hostname smithi053 2024-03-08T05:48:56.029 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i -e 's/smithi053.front.sepia.ceph.com/smithi053/g' /etc/hosts 2024-03-08T05:48:56.061 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i -e 's/172.21.15.53/172.21.15.53/g' /etc/hosts 2024-03-08T05:48:56.133 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i -e 's/smithi053.front.sepia.ceph.com/smithi053/g' /etc/hostname 2024-03-08T05:48:56.205 DEBUG:teuthology.orchestra.run.smithi053:> sudo hostname smithi053 2024-03-08T05:48:56.276 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-03-08T05:48:56.331 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-03-08T05:48:56.331 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="9" 2024-03-08T05:48:56.331 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-03-08T05:48:56.331 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-03-08T05:48:56.331 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="9" 2024-03-08T05:48:56.331 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el9" 2024-03-08T05:48:56.332 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-08T05:48:56.332 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-03-08T05:48:56.332 INFO:teuthology.orchestra.run.smithi053.stdout:LOGO="fedora-logo-icon" 2024-03-08T05:48:56.332 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-08T05:48:56.332 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-03-08T05:48:56.332 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-08T05:48:56.332 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-08T05:48:56.332 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-08T05:48:56.333 INFO:teuthology.provision.fog.smithi053:Deploy complete! 2024-03-08T05:48:56.334 DEBUG:teuthology.parallel:result is None 2024-03-08T06:36:22.594 DEBUG:teuthology.provision.fog.smithi098:Canceling deploy task with ID 839019 2024-03-08T06:36:22.641 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 86, in create self.wait_for_deploy_task(task_id) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 258, in wait_for_deploy_task while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (241) after waiting for 3600 seconds 2024-03-08T06:36:22.642 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-08T06:36:22.927 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=447284510aeb432e8c41b38b2e20f223 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 86, in create self.wait_for_deploy_task(task_id) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 258, in wait_for_deploy_task while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (241) after waiting for 3600 seconds 2024-03-08T06:36:22.929 INFO:teuthology.nuke:Checking targets against current locks 2024-03-08T06:36:22.960 INFO:teuthology.nuke:Not nuking smithi053.front.sepia.ceph.com because description doesn't match: jcollin-2024-03-08_05:34:37-fs:functional-main-distro-default-smithi != /home/teuthworker/archive/jcollin-2024-03-08_05:43:46-fs:functional-main-distro-default-smithi/7585425 2024-03-08T06:36:22.979 INFO:teuthology.nuke:Not nuking smithi098.front.sepia.ceph.com because description doesn't match: jcollin-2024-03-08_05:34:37-fs:functional-main-distro-default-smithi != None 2024-03-08T06:36:22.980 ERROR:teuthology.nuke:Could not nuke the following targets: targets: smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG2m0r2kEKRcX2/qurMpboMLUIsxAqBaY4osw6DmHTnY32NLUpRVYEHZgMMajb5G02msn3VU4kgx3rKGiJMnThU= smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC02tnT2HyLpgg2Trpig9/5XXpkNipXwetAdzme8pyQJiO+Up2+OuVhkqO72eKA0IpeJpOeMPcgc53aNr5ZyL5c= 2024-03-08T06:36:23.214 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 86, in create self.wait_for_deploy_task(task_id) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 258, in wait_for_deploy_task while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (241) after waiting for 3600 seconds