2024-04-01T10:31:19.785 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-01T10:31:19.786 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T10:31:19.995 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6d13c9cc10>, signals=[15]) 2024-04-01T10:31:19.995 INFO:teuthology.lock.ops:Start node 'smithi107.front.sepia.ceph.com' reimaging 2024-04-01T10:31:19.995 INFO:teuthology.lock.ops:Updating [smithi107.front.sepia.ceph.com]: reset os type and version on server 2024-04-01T10:31:19.995 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-04-01T10:31:20.021 INFO:teuthology.lock.ops:Start node 'smithi154.front.sepia.ceph.com' reimaging 2024-04-01T10:31:20.022 INFO:teuthology.lock.ops:Updating [smithi154.front.sepia.ceph.com]: reset os type and version on server 2024-04-01T10:31:20.022 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2024-04-01T10:31:20.046 INFO:teuthology.lock.ops:Start node 'smithi186.front.sepia.ceph.com' reimaging 2024-04-01T10:31:20.046 INFO:teuthology.lock.ops:Updating [smithi186.front.sepia.ceph.com]: reset os type and version on server 2024-04-01T10:31:20.047 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-04-01T10:31:20.140 DEBUG:teuthology.provision.fog.smithi107:Requesting image smithi_centos_9.stream (ID 10) 2024-04-01T10:31:20.170 DEBUG:teuthology.provision.fog.smithi154:Requesting image smithi_centos_9.stream (ID 10) 2024-04-01T10:31:20.189 INFO:teuthology.provision.fog.smithi107:Scheduling deploy of centos 9.stream 2024-04-01T10:31:20.199 DEBUG:teuthology.provision.fog.smithi186:Requesting image smithi_centos_9.stream (ID 10) 2024-04-01T10:31:20.219 INFO:teuthology.provision.fog.smithi154:Scheduling deploy of centos 9.stream 2024-04-01T10:31:20.247 INFO:teuthology.provision.fog.smithi186:Scheduling deploy of centos 9.stream 2024-04-01T10:31:20.416 INFO:teuthology.orchestra.console.smithi107:Power off 2024-04-01T10:31:20.416 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-01T10:31:20.458 INFO:teuthology.orchestra.console.smithi186:Power off 2024-04-01T10:31:20.459 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-01T10:31:20.472 INFO:teuthology.orchestra.console.smithi154:Power off 2024-04-01T10:31:20.472 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-01T10:31:20.479 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2024-04-01T10:31:20.479 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:20.489 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-04-01T10:31:20.489 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:20.498 DEBUG:teuthology.orchestra.console.smithi154:power off output: Chassis Power Control: Down/Off 2024-04-01T10:31:20.498 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:20.507 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-04-01T10:31:20.520 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-01T10:31:20.528 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2024-04-01T10:31:24.510 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:24.521 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:24.529 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:24.620 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-04-01T10:31:24.630 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-01T10:31:24.641 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2024-04-01T10:31:28.623 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:28.637 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:28.644 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:28.740 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-04-01T10:31:28.745 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-04-01T10:31:28.752 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is off 2024-04-01T10:31:28.846 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-04-01T10:31:28.853 INFO:teuthology.orchestra.console.smithi154:Power off completed 2024-04-01T10:31:28.947 INFO:teuthology.orchestra.console.smithi186:Power on 2024-04-01T10:31:28.948 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-01T10:31:28.954 INFO:teuthology.orchestra.console.smithi154:Power on 2024-04-01T10:31:28.954 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-01T10:31:28.967 DEBUG:teuthology.orchestra.console.smithi186:power on output: Chassis Power Control: Up/On 2024-04-01T10:31:28.968 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:28.976 DEBUG:teuthology.orchestra.console.smithi154:power on output: Chassis Power Control: Up/On 2024-04-01T10:31:28.976 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:28.989 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-04-01T10:31:28.996 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is off 2024-04-01T10:31:32.742 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:32.851 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-04-01T10:31:32.951 INFO:teuthology.orchestra.console.smithi107:Power off completed 2024-04-01T10:31:32.991 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:32.997 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:33.052 INFO:teuthology.orchestra.console.smithi107:Power on 2024-04-01T10:31:33.053 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-01T10:31:33.077 DEBUG:teuthology.orchestra.console.smithi107:power on output: Chassis Power Control: Up/On 2024-04-01T10:31:33.078 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:33.099 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-04-01T10:31:33.101 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-04-01T10:31:33.106 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is off 2024-04-01T10:31:37.102 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:37.111 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:37.118 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T10:31:37.213 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-01T10:31:37.220 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-04-01T10:31:37.227 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2024-04-01T10:31:37.314 INFO:teuthology.orchestra.console.smithi186:Power on completed 2024-04-01T10:31:37.321 INFO:teuthology.orchestra.console.smithi107:Power on completed 2024-04-01T10:31:37.328 INFO:teuthology.orchestra.console.smithi154:Power on completed 2024-04-01T10:31:37.415 INFO:teuthology.provision.fog.smithi186:Waiting for deploy to finish 2024-04-01T10:31:37.422 INFO:teuthology.provision.fog.smithi107:Waiting for deploy to finish 2024-04-01T10:31:37.428 INFO:teuthology.provision.fog.smithi154:Waiting for deploy to finish 2024-04-01T10:34:23.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:34:23.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:34:23.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:34:26.466 WARNING:teuthology.provision.fog.smithi154:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-04-01T10:34:26.494 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-01T10:34:26.495 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-01T10:34:32.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:34:32.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:34:32.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:34:35.550 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-01T10:34:35.551 WARNING:teuthology.provision.fog.smithi154:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-04-01T10:34:35.551 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-01T10:34:41.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:34:41.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:34:41.559 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:34:44.610 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-01T10:34:44.638 WARNING:teuthology.provision.fog.smithi154:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-04-01T10:34:44.639 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-01T10:34:50.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:34:50.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:34:50.642 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:34:53.698 WARNING:teuthology.provision.fog.smithi154:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-04-01T10:34:53.699 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-01T10:34:53.699 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-01T10:34:59.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:34:59.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:34:59.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:35:02.782 WARNING:teuthology.provision.fog.smithi154:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-04-01T10:35:02.783 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-01T10:35:02.786 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-01T10:35:08.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:35:08.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:35:08.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:35:11.842 WARNING:teuthology.provision.fog.smithi154:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-04-01T10:35:11.843 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-01T10:35:11.874 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-01T10:35:17.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:35:17.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:35:17.878 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:35:20.926 WARNING:teuthology.provision.fog.smithi154:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-04-01T10:35:26.931 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:35:29.982 WARNING:teuthology.provision.fog.smithi154:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-04-01T10:35:35.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:35:36.350 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-01T10:35:36.351 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-01T10:35:42.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:35:42.357 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:35:45.406 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-01T10:35:45.438 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-01T10:35:51.411 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:35:51.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:35:54.462 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-01T10:35:54.494 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-01T10:36:00.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:36:00.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:36:00.769 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-04-01T10:36:00.769 DEBUG:teuthology.orchestra.run.smithi107:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-04-01T10:36:00.778 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-04-01T10:36:00.778 DEBUG:teuthology.orchestra.run.smithi186:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-04-01T10:36:16.205 INFO:teuthology.provision.fog.smithi107:Node is ready 2024-04-01T10:36:16.205 DEBUG:teuthology.orchestra.run.smithi107:> hostname 2024-04-01T10:36:16.222 INFO:teuthology.orchestra.run.smithi107.stdout:smithi046 2024-04-01T10:36:16.223 DEBUG:teuthology.orchestra.run.smithi107:> grep smithi046 /etc/hosts 2024-04-01T10:36:16.280 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.46 smithi046.front.sepia.ceph.com smithi046 2024-04-01T10:36:16.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname smithi107 2024-04-01T10:36:16.307 INFO:teuthology.provision.fog.smithi186:Node is ready 2024-04-01T10:36:16.307 DEBUG:teuthology.orchestra.run.smithi186:> hostname 2024-04-01T10:36:16.324 INFO:teuthology.orchestra.run.smithi186.stdout:smithi046 2024-04-01T10:36:16.324 DEBUG:teuthology.orchestra.run.smithi186:> grep smithi046 /etc/hosts 2024-04-01T10:36:16.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T10:36:16.382 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i -e 's/smithi046/smithi186/g' /etc/hostname 2024-04-01T10:36:16.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/smithi046/smithi107/g' /etc/hosts 2024-04-01T10:36:16.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/172.21.15.46/172.21.15.107/g' /etc/hosts 2024-04-01T10:36:16.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/smithi046/smithi107/g' /etc/hostname 2024-04-01T10:36:16.529 DEBUG:teuthology.orchestra.run.smithi186:> sudo hostname smithi186 2024-04-01T10:36:16.556 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-04-01T10:36:16.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname smithi107 2024-04-01T10:36:16.611 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-04-01T10:36:16.612 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-04-01T10:36:16.612 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-04-01T10:36:16.612 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-04-01T10:36:16.612 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-04-01T10:36:16.612 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-04-01T10:36:16.612 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T10:36:16.612 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-04-01T10:36:16.612 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-04-01T10:36:16.612 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T10:36:16.612 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-04-01T10:36:16.612 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T10:36:16.612 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T10:36:16.613 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T10:36:16.613 INFO:teuthology.provision.fog.smithi186:Deploy complete! 2024-04-01T10:36:16.615 DEBUG:teuthology.parallel:result is None 2024-04-01T10:36:16.652 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-04-01T10:36:16.707 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-04-01T10:36:16.708 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-04-01T10:36:16.708 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-04-01T10:36:16.708 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-04-01T10:36:16.708 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-04-01T10:36:16.708 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-04-01T10:36:16.708 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T10:36:16.708 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-04-01T10:36:16.708 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-04-01T10:36:16.708 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T10:36:16.708 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-04-01T10:36:16.708 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T10:36:16.708 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T10:36:16.709 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T10:36:16.709 INFO:teuthology.provision.fog.smithi107:Deploy complete! 2024-04-01T10:36:16.709 DEBUG:teuthology.parallel:result is None 2024-04-01T10:36:36.002 WARNING:teuthology.provision.fog.smithi154:timed out 2024-04-01T10:36:42.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:36:42.256 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-04-01T10:36:42.256 DEBUG:teuthology.orchestra.run.smithi154:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-04-01T10:36:42.583 INFO:teuthology.provision.fog.smithi154:Node is ready 2024-04-01T10:36:42.584 DEBUG:teuthology.orchestra.run.smithi154:> hostname 2024-04-01T10:36:42.598 INFO:teuthology.orchestra.run.smithi154.stdout:smithi154.front.sepia.ceph.com 2024-04-01T10:36:42.598 DEBUG:teuthology.orchestra.run.smithi154:> grep smithi154.front.sepia.ceph.com /etc/hosts 2024-04-01T10:36:42.654 INFO:teuthology.orchestra.run.smithi154.stdout:172.21.15.154 smithi154.front.sepia.ceph.com smithi154 2024-04-01T10:36:42.655 DEBUG:teuthology.orchestra.run.smithi154:> sudo hostname smithi154 2024-04-01T10:36:42.758 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i -e 's/smithi154.front.sepia.ceph.com/smithi154/g' /etc/hosts 2024-04-01T10:36:42.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i -e 's/172.21.15.154/172.21.15.154/g' /etc/hosts 2024-04-01T10:36:42.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i -e 's/smithi154.front.sepia.ceph.com/smithi154/g' /etc/hostname 2024-04-01T10:36:42.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo hostname smithi154 2024-04-01T10:36:42.988 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2024-04-01T10:36:43.040 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="CentOS Stream" 2024-04-01T10:36:43.040 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="9" 2024-04-01T10:36:43.040 INFO:teuthology.orchestra.run.smithi154.stdout:ID="centos" 2024-04-01T10:36:43.040 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="rhel fedora" 2024-04-01T10:36:43.040 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="9" 2024-04-01T10:36:43.040 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el9" 2024-04-01T10:36:43.041 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T10:36:43.041 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2024-04-01T10:36:43.041 INFO:teuthology.orchestra.run.smithi154.stdout:LOGO="fedora-logo-icon" 2024-04-01T10:36:43.041 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T10:36:43.041 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://centos.org/" 2024-04-01T10:36:43.041 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T10:36:43.041 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T10:36:43.041 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T10:36:43.041 INFO:teuthology.provision.fog.smithi154:Deploy complete! 2024-04-01T10:36:43.042 DEBUG:teuthology.parallel:result is None 2024-04-01T10:36:43.260 INFO:teuthology.lock.ops:Checking smithi107.front.sepia.ceph.com 2024-04-01T10:36:43.261 INFO:teuthology.lock.ops:New key found. Updating... 2024-04-01T10:36:43.285 INFO:teuthology.lock.ops:Checking smithi154.front.sepia.ceph.com 2024-04-01T10:36:43.285 INFO:teuthology.lock.ops:New key found. Updating... 2024-04-01T10:36:43.309 INFO:teuthology.lock.ops:Checking smithi186.front.sepia.ceph.com 2024-04-01T10:36:43.309 INFO:teuthology.lock.ops:New key found. Updating... 2024-04-01T10:36:43.327 INFO:teuthology.lock.ops:Updating [smithi107.front.sepia.ceph.com]: set os type and version on server 2024-04-01T10:36:43.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-01T10:36:43.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:36:43.419 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-04-01T10:36:43.592 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-01T10:36:43.592 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-04-01T10:36:43.652 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-04-01T10:36:43.653 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-04-01T10:36:43.707 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-04-01T10:36:43.707 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-04-01T10:36:43.707 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-04-01T10:36:43.707 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-04-01T10:36:43.707 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-04-01T10:36:43.708 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-04-01T10:36:43.708 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T10:36:43.708 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-04-01T10:36:43.708 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-04-01T10:36:43.708 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T10:36:43.708 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-04-01T10:36:43.708 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T10:36:43.708 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T10:36:43.708 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T10:36:43.709 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-04-01T10:36:43.735 INFO:teuthology.lock.ops:Updating [smithi154.front.sepia.ceph.com]: set os type and version on server 2024-04-01T10:36:43.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-04-01T10:36:43.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:36:43.827 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-04-01T10:36:43.899 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-04-01T10:36:43.899 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2024-04-01T10:36:43.953 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2024-04-01T10:36:43.954 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2024-04-01T10:36:44.007 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="CentOS Stream" 2024-04-01T10:36:44.007 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="9" 2024-04-01T10:36:44.007 INFO:teuthology.orchestra.run.smithi154.stdout:ID="centos" 2024-04-01T10:36:44.007 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="rhel fedora" 2024-04-01T10:36:44.007 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="9" 2024-04-01T10:36:44.007 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el9" 2024-04-01T10:36:44.007 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T10:36:44.007 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2024-04-01T10:36:44.007 INFO:teuthology.orchestra.run.smithi154.stdout:LOGO="fedora-logo-icon" 2024-04-01T10:36:44.007 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T10:36:44.008 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://centos.org/" 2024-04-01T10:36:44.008 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T10:36:44.008 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T10:36:44.008 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T10:36:44.008 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2024-04-01T10:36:44.034 INFO:teuthology.lock.ops:Updating [smithi186.front.sepia.ceph.com]: set os type and version on server 2024-04-01T10:36:44.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-01T10:36:44.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T10:36:44.128 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-04-01T10:36:44.301 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-01T10:36:44.301 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-04-01T10:36:44.356 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-04-01T10:36:44.356 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-04-01T10:36:44.412 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-04-01T10:36:44.413 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-04-01T10:36:44.413 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-04-01T10:36:44.413 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-04-01T10:36:44.413 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-04-01T10:36:44.413 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-04-01T10:36:44.413 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T10:36:44.413 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-04-01T10:36:44.413 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-04-01T10:36:44.413 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T10:36:44.413 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-04-01T10:36:44.413 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T10:36:44.413 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T10:36:44.414 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T10:36:44.414 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-04-01T10:36:44.434 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T10:36:44.525 INFO:teuthology.dispatcher.supervisor:Running job 7626077 2024-04-01T10:36:44.525 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology -v --owner scheduled_pdiazbou@teuthology --archive /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626077 --name pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi --description rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} -- /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626077/orig.config.yaml 2024-04-01T10:36:44.528 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626077 2024-04-01T10:36:44.528 INFO:teuthology.dispatcher.supervisor:Job PID: 1976532 2024-04-01T10:36:44.528 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2024-04-01T10:38:44.627 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T10:40:44.801 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T10:42:44.949 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T10:44:45.107 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T10:46:45.287 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T10:48:45.463 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T10:50:45.635 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T10:50:45.725 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1