2024-03-02T04:37:54.823 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-03-02T04:37:54.824 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-02T04:37:54.986 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc94d040c10>, signals=[15]) 2024-03-02T04:37:54.986 INFO:teuthology.lock.ops:Start node 'smithi053.front.sepia.ceph.com' reimaging 2024-03-02T04:37:54.986 INFO:teuthology.lock.ops:Updating [smithi053.front.sepia.ceph.com]: reset os type and version on server 2024-03-02T04:37:54.987 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-03-02T04:37:55.009 INFO:teuthology.lock.ops:Start node 'smithi064.front.sepia.ceph.com' reimaging 2024-03-02T04:37:55.009 INFO:teuthology.lock.ops:Updating [smithi064.front.sepia.ceph.com]: reset os type and version on server 2024-03-02T04:37:55.009 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-03-02T04:37:55.034 INFO:teuthology.lock.ops:Start node 'smithi184.front.sepia.ceph.com' reimaging 2024-03-02T04:37:55.034 INFO:teuthology.lock.ops:Updating [smithi184.front.sepia.ceph.com]: reset os type and version on server 2024-03-02T04:37:55.034 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2024-03-02T04:37:55.129 DEBUG:teuthology.provision.fog.smithi053:Requesting image smithi_centos_8.stream (ID 3) 2024-03-02T04:37:55.134 DEBUG:teuthology.provision.fog.smithi064:Requesting image smithi_centos_8.stream (ID 3) 2024-03-02T04:37:55.161 DEBUG:teuthology.provision.fog.smithi184:Requesting image smithi_centos_8.stream (ID 3) 2024-03-02T04:37:55.173 INFO:teuthology.provision.fog.smithi064:Scheduling deploy of centos 8.stream 2024-03-02T04:37:55.196 INFO:teuthology.provision.fog.smithi053:Scheduling deploy of centos 8.stream 2024-03-02T04:37:55.220 INFO:teuthology.provision.fog.smithi184:Scheduling deploy of centos 8.stream 2024-03-02T04:37:55.417 INFO:teuthology.orchestra.console.smithi064:Power off 2024-03-02T04:37:55.417 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-02T04:37:55.436 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2024-03-02T04:37:55.437 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T04:37:55.458 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-03-02T04:37:55.475 INFO:teuthology.orchestra.console.smithi053:Power off 2024-03-02T04:37:55.475 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-02T04:37:55.488 INFO:teuthology.orchestra.console.smithi184:Power off 2024-03-02T04:37:55.489 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-02T04:37:55.498 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-03-02T04:37:55.498 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T04:37:55.509 DEBUG:teuthology.orchestra.console.smithi184:power off output: Chassis Power Control: Down/Off 2024-03-02T04:37:55.509 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T04:37:55.519 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-02T04:37:55.529 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is off 2024-03-02T04:37:55.558 INFO:teuthology.orchestra.console.smithi064:Power off completed 2024-03-02T04:37:55.620 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-03-02T04:37:55.630 INFO:teuthology.orchestra.console.smithi184:Power off completed 2024-03-02T04:37:55.660 INFO:teuthology.orchestra.console.smithi064:Power on 2024-03-02T04:37:55.660 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-02T04:37:55.681 DEBUG:teuthology.orchestra.console.smithi064:power on output: Chassis Power Control: Up/On 2024-03-02T04:37:55.681 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T04:37:55.721 INFO:teuthology.orchestra.console.smithi053:Power on 2024-03-02T04:37:55.721 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-02T04:37:55.732 INFO:teuthology.orchestra.console.smithi184:Power on 2024-03-02T04:37:55.732 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-02T04:37:55.741 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-03-02T04:37:55.746 DEBUG:teuthology.orchestra.console.smithi053:power on output: Chassis Power Control: Up/On 2024-03-02T04:37:55.746 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T04:37:55.759 DEBUG:teuthology.orchestra.console.smithi184:power on output: Chassis Power Control: Up/On 2024-03-02T04:37:55.759 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T04:37:55.768 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-02T04:37:55.782 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is off 2024-03-02T04:37:59.744 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T04:37:59.769 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T04:37:59.783 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T04:37:59.854 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-03-02T04:37:59.878 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-02T04:37:59.891 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-03-02T04:37:59.992 INFO:teuthology.orchestra.console.smithi184:Power on completed 2024-03-02T04:38:00.093 INFO:teuthology.provision.fog.smithi184:Waiting for deploy to finish 2024-03-02T04:38:03.856 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T04:38:03.879 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T04:38:03.988 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-02T04:38:03.989 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-02T04:38:04.089 INFO:teuthology.orchestra.console.smithi053:Power on completed 2024-03-02T04:38:04.090 INFO:teuthology.orchestra.console.smithi064:Power on completed 2024-03-02T04:38:04.190 INFO:teuthology.provision.fog.smithi053:Waiting for deploy to finish 2024-03-02T04:38:04.192 INFO:teuthology.provision.fog.smithi064:Waiting for deploy to finish 2024-03-02T04:40:35.037 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-02T04:40:38.112 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-02T04:40:44.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-02T04:40:47.196 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-02T04:40:50.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-02T04:40:53.180 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-02T04:40:53.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-02T04:40:56.252 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-02T04:40:59.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-02T04:41:02.240 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-02T04:41:02.253 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-02T04:41:05.308 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-02T04:41:08.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-02T04:41:11.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-02T04:41:11.328 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-02T04:41:17.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-02T04:41:20.384 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-02T04:41:26.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-02T04:41:29.468 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-02T04:41:35.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-02T04:42:11.321 WARNING:teuthology.provision.fog.smithi053:timed out 2024-03-02T04:42:17.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-02T04:42:17.580 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-02T04:42:17.580 DEBUG:teuthology.orchestra.run.smithi053:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-02T04:42:17.814 INFO:teuthology.provision.fog.smithi053:Node is ready 2024-03-02T04:42:17.814 DEBUG:teuthology.orchestra.run.smithi053:> hostname 2024-03-02T04:42:17.836 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053.front.sepia.ceph.com 2024-03-02T04:42:17.836 DEBUG:teuthology.orchestra.run.smithi053:> grep smithi053.front.sepia.ceph.com /etc/hosts 2024-03-02T04:42:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53 smithi053.front.sepia.ceph.com smithi053 2024-03-02T04:42:17.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo hostname smithi053 2024-03-02T04:42:17.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i -e 's/smithi053.front.sepia.ceph.com/smithi053/g' /etc/hosts 2024-03-02T04:42:18.020 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i -e 's/172.21.15.53/172.21.15.53/g' /etc/hosts 2024-03-02T04:42:18.091 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i -e 's/smithi053.front.sepia.ceph.com/smithi053/g' /etc/hostname 2024-03-02T04:42:18.161 DEBUG:teuthology.orchestra.run.smithi053:> sudo hostname smithi053 2024-03-02T04:42:18.231 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-03-02T04:42:18.291 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-03-02T04:42:18.291 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="8" 2024-03-02T04:42:18.292 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-03-02T04:42:18.292 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-03-02T04:42:18.292 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="8" 2024-03-02T04:42:18.292 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el8" 2024-03-02T04:42:18.292 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-02T04:42:18.292 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-03-02T04:42:18.292 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-02T04:42:18.292 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-03-02T04:42:18.292 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-02T04:42:18.292 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-02T04:42:18.292 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-02T04:42:18.293 INFO:teuthology.provision.fog.smithi053:Deploy complete! 2024-03-02T04:42:18.294 DEBUG:teuthology.parallel:result is None 2024-03-02T04:42:35.476 WARNING:teuthology.provision.fog.smithi064:timed out 2024-03-02T04:42:41.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-02T04:42:41.727 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-02T04:42:41.727 DEBUG:teuthology.orchestra.run.smithi064:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-02T04:42:41.978 INFO:teuthology.provision.fog.smithi064:Node is ready 2024-03-02T04:42:41.978 DEBUG:teuthology.orchestra.run.smithi064:> hostname 2024-03-02T04:42:42.039 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064.front.sepia.ceph.com 2024-03-02T04:42:42.040 DEBUG:teuthology.orchestra.run.smithi064:> grep smithi064.front.sepia.ceph.com /etc/hosts 2024-03-02T04:42:42.101 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64 smithi064.front.sepia.ceph.com smithi064 2024-03-02T04:42:42.101 DEBUG:teuthology.orchestra.run.smithi064:> sudo hostname smithi064 2024-03-02T04:42:42.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i -e 's/smithi064.front.sepia.ceph.com/smithi064/g' /etc/hosts 2024-03-02T04:42:42.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i -e 's/172.21.15.64/172.21.15.64/g' /etc/hosts 2024-03-02T04:42:42.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i -e 's/smithi064.front.sepia.ceph.com/smithi064/g' /etc/hostname 2024-03-02T04:42:42.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo hostname smithi064 2024-03-02T04:42:42.438 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-03-02T04:42:42.496 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-03-02T04:42:42.497 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="8" 2024-03-02T04:42:42.497 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-03-02T04:42:42.497 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-03-02T04:42:42.497 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="8" 2024-03-02T04:42:42.497 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el8" 2024-03-02T04:42:42.497 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-02T04:42:42.497 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-03-02T04:42:42.497 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-02T04:42:42.497 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-03-02T04:42:42.497 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-02T04:42:42.497 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-02T04:42:42.497 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-02T04:42:42.498 INFO:teuthology.provision.fog.smithi064:Deploy complete! 2024-03-02T04:42:42.498 DEBUG:teuthology.parallel:result is None 2024-03-02T05:38:04.915 DEBUG:teuthology.provision.fog.smithi184:Canceling deploy task with ID 826428 2024-03-02T05:38:04.983 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/fog.py", line 86, in create self.wait_for_deploy_task(task_id) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/fog.py", line 258, in wait_for_deploy_task while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (241) after waiting for 3600 seconds 2024-03-02T05:38:04.985 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-02T05:38:05.276 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1ab08f92fff74667b83681270331c542 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/fog.py", line 86, in create self.wait_for_deploy_task(task_id) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/fog.py", line 258, in wait_for_deploy_task while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (241) after waiting for 3600 seconds 2024-03-02T05:38:05.278 INFO:teuthology.nuke:Checking targets against current locks 2024-03-02T05:38:05.334 DEBUG:teuthology.nuke:shortname: smithi053 2024-03-02T05:38:05.334 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-02T05:38:05.363 DEBUG:teuthology.nuke:shortname: smithi064 2024-03-02T05:38:05.363 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-02T05:38:05.372 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578665', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-02 04:37:53.712141', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGlNrO950NDOTnnQbUrZtOS/JAReZXEztW2pn2Tfw8TKK/zPHksbaKffZIsSih4tJY+3u4KCGDIP47hFl8RQmuA='} 2024-03-02T05:38:05.390 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578665', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-02 04:37:53.713316', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGhfM2YZCQSk6usspFAU31XZ47p4bnnOA4t1j0JDJHDFMSfGv9jOw3ql2ajn5nOoXOUV41cdOeJzCm7spa8S3F8='} 2024-03-02T05:38:05.396 DEBUG:teuthology.nuke:shortname: smithi184 2024-03-02T05:38:05.396 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-02T05:38:05.419 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578665', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-02 04:37:53.714252', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCbP9v7ZEJ0vOo+wt6LS/VcY8VI6ZZWgaw9qMEVkeh0yJmfJ7GaXtR4Jtem9tQH0HcpwKOBu4GT8guLZAaphISA='} 2024-03-02T05:38:05.428 INFO:teuthology.orchestra.console.smithi053:Power off 2024-03-02T05:38:05.431 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-02T05:38:05.459 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-03-02T05:38:05.459 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T05:38:05.478 INFO:teuthology.orchestra.console.smithi064:Power off 2024-03-02T05:38:05.479 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-02T05:38:05.488 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-02T05:38:05.501 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2024-03-02T05:38:05.501 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T05:38:05.520 INFO:teuthology.orchestra.console.smithi184:Power off 2024-03-02T05:38:05.520 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-02T05:38:05.529 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-02T05:38:05.541 DEBUG:teuthology.orchestra.console.smithi184:power off output: Chassis Power Control: Down/Off 2024-03-02T05:38:05.541 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T05:38:05.563 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is off 2024-03-02T05:38:05.664 INFO:teuthology.orchestra.console.smithi184:Power off completed 2024-03-02T05:38:05.813 INFO:teuthology.lock.ops:unlocked: smithi184.front.sepia.ceph.com 2024-03-02T05:38:09.492 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T05:38:09.530 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T05:38:09.603 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-02T05:38:09.641 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-02T05:38:13.608 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T05:38:13.643 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T05:38:13.718 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-02T05:38:13.753 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-02T05:38:13.819 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-03-02T05:38:13.977 INFO:teuthology.lock.ops:unlocked: smithi053.front.sepia.ceph.com 2024-03-02T05:38:17.756 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-02T05:38:17.865 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-03-02T05:38:17.966 INFO:teuthology.orchestra.console.smithi064:Power off completed 2024-03-02T05:38:18.120 INFO:teuthology.lock.ops:unlocked: smithi064.front.sepia.ceph.com 2024-03-02T05:38:18.478 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/fog.py", line 86, in create self.wait_for_deploy_task(task_id) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/fog.py", line 258, in wait_for_deploy_task while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (241) after waiting for 3600 seconds