2024-03-07T13:36:43.294 INFO:root:teuthology version: 0.0.1.dev262+ge691533
2024-03-07T13:36:43.294 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:36:43.438 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f1173d89f40>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f1174ceac10>, signals=[15])
2024-03-07T13:36:43.438 INFO:teuthology.lock.ops:Start node 'smithi098.front.sepia.ceph.com' reimaging
2024-03-07T13:36:43.439 INFO:teuthology.lock.ops:Updating [smithi098.front.sepia.ceph.com]: reset os type and version on server
2024-03-07T13:36:43.439 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server
2024-03-07T13:36:43.463 INFO:teuthology.lock.ops:Start node 'smithi195.front.sepia.ceph.com' reimaging
2024-03-07T13:36:43.463 INFO:teuthology.lock.ops:Updating [smithi195.front.sepia.ceph.com]: reset os type and version on server
2024-03-07T13:36:43.463 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server
2024-03-07T13:36:43.585 DEBUG:teuthology.provision.fog.smithi098:Requesting image smithi_centos_9.stream (ID 10)
2024-03-07T13:36:43.602 DEBUG:teuthology.provision.fog.smithi195:Requesting image smithi_centos_9.stream (ID 10)
2024-03-07T13:36:43.634 INFO:teuthology.provision.fog.smithi098:Scheduling deploy of centos 9.stream
2024-03-07T13:36:43.643 INFO:teuthology.provision.fog.smithi195:Scheduling deploy of centos 9.stream
2024-03-07T13:36:43.904 INFO:teuthology.orchestra.console.smithi195:Power off
2024-03-07T13:36:43.905 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-07T13:36:43.918 INFO:teuthology.orchestra.console.smithi098:Power off
2024-03-07T13:36:43.919 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-07T13:36:43.928 DEBUG:teuthology.orchestra.console.smithi195:power off output: Chassis Power Control: Down/Off
2024-03-07T13:36:43.929 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:36:43.942 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off
2024-03-07T13:36:43.943 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:36:43.953 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on
2024-03-07T13:36:43.965 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on
2024-03-07T13:36:47.956 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:36:47.967 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:36:48.065 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on
2024-03-07T13:36:48.077 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on
2024-03-07T13:36:52.070 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:36:52.077 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:36:52.180 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on
2024-03-07T13:36:52.189 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off
2024-03-07T13:36:52.290 INFO:teuthology.orchestra.console.smithi098:Power off completed
2024-03-07T13:36:52.391 INFO:teuthology.orchestra.console.smithi098:Power on
2024-03-07T13:36:52.391 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-07T13:36:52.412 DEBUG:teuthology.orchestra.console.smithi098:power on output: Chassis Power Control: Up/On
2024-03-07T13:36:52.412 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:36:52.464 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off
2024-03-07T13:36:56.184 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:36:56.294 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off
2024-03-07T13:36:56.395 INFO:teuthology.orchestra.console.smithi195:Power off completed
2024-03-07T13:36:56.465 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:36:56.496 INFO:teuthology.orchestra.console.smithi195:Power on
2024-03-07T13:36:56.496 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-07T13:36:56.547 DEBUG:teuthology.orchestra.console.smithi195:power on output: Chassis Power Control: Up/On
2024-03-07T13:36:56.548 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:36:56.572 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off
2024-03-07T13:36:56.575 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off
2024-03-07T13:37:00.577 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:00.585 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:00.687 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off
2024-03-07T13:37:00.694 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on
2024-03-07T13:37:00.795 INFO:teuthology.orchestra.console.smithi098:Power on completed
2024-03-07T13:37:00.896 INFO:teuthology.provision.fog.smithi098:Waiting for deploy to finish
2024-03-07T13:37:04.692 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:04.835 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on
2024-03-07T13:37:04.936 INFO:teuthology.orchestra.console.smithi195:Power on completed
2024-03-07T13:37:05.037 INFO:teuthology.provision.fog.smithi195:Waiting for deploy to finish
2024-03-07T13:39:31.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:39:34.976 WARNING:teuthology.provision.fog.smithi098:[Errno None] Unable to connect to port 22 on 172.21.15.98
2024-03-07T13:39:35.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:39:39.068 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195
2024-03-07T13:39:40.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:39:44.060 WARNING:teuthology.provision.fog.smithi098:[Errno None] Unable to connect to port 22 on 172.21.15.98
2024-03-07T13:39:45.070 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:39:48.124 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195
2024-03-07T13:39:50.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:39:53.116 WARNING:teuthology.provision.fog.smithi098:[Errno None] Unable to connect to port 22 on 172.21.15.98
2024-03-07T13:39:54.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:39:57.180 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195
2024-03-07T13:39:59.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:02.172 WARNING:teuthology.provision.fog.smithi098:[Errno None] Unable to connect to port 22 on 172.21.15.98
2024-03-07T13:40:03.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:06.236 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195
2024-03-07T13:40:08.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:11.228 WARNING:teuthology.provision.fog.smithi098:[Errno None] Unable to connect to port 22 on 172.21.15.98
2024-03-07T13:40:12.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:15.292 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195
2024-03-07T13:40:17.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:20.284 WARNING:teuthology.provision.fog.smithi098:[Errno None] Unable to connect to port 22 on 172.21.15.98
2024-03-07T13:40:21.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:24.352 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195
2024-03-07T13:40:26.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:29.340 WARNING:teuthology.provision.fog.smithi098:[Errno None] Unable to connect to port 22 on 172.21.15.98
2024-03-07T13:40:30.354 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:35.345 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:40.636 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195
2024-03-07T13:40:46.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:49.692 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195
2024-03-07T13:40:55.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:58.780 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195
2024-03-07T13:41:04.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:07.223 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-07T13:41:07.224 DEBUG:teuthology.orchestra.run.smithi195:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-07T13:41:27.543 INFO:teuthology.provision.fog.smithi195:Node is ready
2024-03-07T13:41:27.544 DEBUG:teuthology.orchestra.run.smithi195:> hostname
2024-03-07T13:41:27.563 INFO:teuthology.orchestra.run.smithi195.stdout:smithi195.front.sepia.ceph.com
2024-03-07T13:41:27.563 DEBUG:teuthology.orchestra.run.smithi195:> grep smithi195.front.sepia.ceph.com /etc/hosts
2024-03-07T13:41:27.620 INFO:teuthology.orchestra.run.smithi195.stdout:172.21.15.195 smithi195.front.sepia.ceph.com smithi195
2024-03-07T13:41:27.620 DEBUG:teuthology.orchestra.run.smithi195:> sudo hostname smithi195
2024-03-07T13:41:27.737 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i -e 's/smithi195.front.sepia.ceph.com/smithi195/g' /etc/hosts
2024-03-07T13:41:27.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i -e 's/172.21.15.195/172.21.15.195/g' /etc/hosts
2024-03-07T13:41:27.836 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i -e 's/smithi195.front.sepia.ceph.com/smithi195/g' /etc/hostname
2024-03-07T13:41:27.906 DEBUG:teuthology.orchestra.run.smithi195:> sudo hostname smithi195
2024-03-07T13:41:27.975 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release
2024-03-07T13:41:28.032 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream"
2024-03-07T13:41:28.032 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="9"
2024-03-07T13:41:28.032 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos"
2024-03-07T13:41:28.032 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora"
2024-03-07T13:41:28.032 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="9"
2024-03-07T13:41:28.032 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el9"
2024-03-07T13:41:28.032 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-07T13:41:28.032 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31"
2024-03-07T13:41:28.032 INFO:teuthology.orchestra.run.smithi195.stdout:LOGO="fedora-logo-icon"
2024-03-07T13:41:28.032 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-07T13:41:28.032 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/"
2024-03-07T13:41:28.032 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-07T13:41:28.032 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-07T13:41:28.032 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-07T13:41:28.033 INFO:teuthology.provision.fog.smithi195:Deploy complete!
2024-03-07T13:41:28.034 DEBUG:teuthology.parallel:result is None
2024-03-07T13:41:35.352 WARNING:teuthology.provision.fog.smithi098:timed out
2024-03-07T13:41:41.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:41.577 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-07T13:41:41.577 DEBUG:teuthology.orchestra.run.smithi098:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-07T13:41:41.888 INFO:teuthology.provision.fog.smithi098:Node is ready
2024-03-07T13:41:41.888 DEBUG:teuthology.orchestra.run.smithi098:> hostname
2024-03-07T13:41:41.902 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098.front.sepia.ceph.com
2024-03-07T13:41:41.902 DEBUG:teuthology.orchestra.run.smithi098:> grep smithi098.front.sepia.ceph.com /etc/hosts
2024-03-07T13:41:41.959 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98 smithi098.front.sepia.ceph.com smithi098
2024-03-07T13:41:41.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname smithi098
2024-03-07T13:41:42.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i -e 's/smithi098.front.sepia.ceph.com/smithi098/g' /etc/hosts
2024-03-07T13:41:42.099 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i -e 's/172.21.15.98/172.21.15.98/g' /etc/hosts
2024-03-07T13:41:42.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i -e 's/smithi098.front.sepia.ceph.com/smithi098/g' /etc/hostname
2024-03-07T13:41:42.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname smithi098
2024-03-07T13:41:42.305 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release
2024-03-07T13:41:42.361 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream"
2024-03-07T13:41:42.362 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9"
2024-03-07T13:41:42.362 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos"
2024-03-07T13:41:42.362 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora"
2024-03-07T13:41:42.362 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9"
2024-03-07T13:41:42.362 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9"
2024-03-07T13:41:42.362 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-07T13:41:42.362 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31"
2024-03-07T13:41:42.362 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon"
2024-03-07T13:41:42.362 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-07T13:41:42.362 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/"
2024-03-07T13:41:42.362 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-07T13:41:42.362 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-07T13:41:42.362 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-07T13:41:42.363 INFO:teuthology.provision.fog.smithi098:Deploy complete!
2024-03-07T13:41:42.363 DEBUG:teuthology.parallel:result is None
2024-03-07T13:41:42.530 INFO:teuthology.lock.ops:Checking smithi098.front.sepia.ceph.com
2024-03-07T13:41:42.530 INFO:teuthology.lock.ops:New key found. Updating...
2024-03-07T13:41:42.549 INFO:teuthology.lock.ops:Checking smithi195.front.sepia.ceph.com
2024-03-07T13:41:42.549 INFO:teuthology.lock.ops:New key found. Updating...
2024-03-07T13:41:42.566 INFO:teuthology.lock.ops:Updating [smithi098.front.sepia.ceph.com]: set os type and version on server
2024-03-07T13:41:42.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com'
2024-03-07T13:41:42.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:42.652 DEBUG:teuthology.orchestra.run.smithi098:> true
2024-03-07T13:41:42.718 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com'
2024-03-07T13:41:42.719 DEBUG:teuthology.orchestra.run.smithi098:> uname -m
2024-03-07T13:41:42.772 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64
2024-03-07T13:41:42.773 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release
2024-03-07T13:41:42.827 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream"
2024-03-07T13:41:42.828 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9"
2024-03-07T13:41:42.828 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos"
2024-03-07T13:41:42.828 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora"
2024-03-07T13:41:42.828 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9"
2024-03-07T13:41:42.828 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9"
2024-03-07T13:41:42.828 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-07T13:41:42.828 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31"
2024-03-07T13:41:42.828 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon"
2024-03-07T13:41:42.828 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-07T13:41:42.828 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/"
2024-03-07T13:41:42.828 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-07T13:41:42.829 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-07T13:41:42.829 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-07T13:41:42.829 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server
2024-03-07T13:41:42.849 INFO:teuthology.lock.ops:Updating [smithi195.front.sepia.ceph.com]: set os type and version on server
2024-03-07T13:41:42.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com'
2024-03-07T13:41:42.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:42.936 DEBUG:teuthology.orchestra.run.smithi195:> true
2024-03-07T13:41:43.107 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com'
2024-03-07T13:41:43.107 DEBUG:teuthology.orchestra.run.smithi195:> uname -m
2024-03-07T13:41:43.162 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64
2024-03-07T13:41:43.162 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release
2024-03-07T13:41:43.216 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream"
2024-03-07T13:41:43.216 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="9"
2024-03-07T13:41:43.216 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos"
2024-03-07T13:41:43.217 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora"
2024-03-07T13:41:43.217 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="9"
2024-03-07T13:41:43.217 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el9"
2024-03-07T13:41:43.217 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-07T13:41:43.217 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31"
2024-03-07T13:41:43.217 INFO:teuthology.orchestra.run.smithi195.stdout:LOGO="fedora-logo-icon"
2024-03-07T13:41:43.217 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-07T13:41:43.217 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/"
2024-03-07T13:41:43.217 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-07T13:41:43.217 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-07T13:41:43.217 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-07T13:41:43.217 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server
2024-03-07T13:41:43.240 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:41:43.342 INFO:teuthology.dispatcher.supervisor:Running job 7584479
2024-03-07T13:41:43.343 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7584479 --name vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi --description fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} -- /home/teuthworker/archive/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7584479/orig.config.yaml
2024-03-07T13:41:43.346 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7584479
2024-03-07T13:41:43.346 INFO:teuthology.dispatcher.supervisor:Job PID: 690075
2024-03-07T13:41:43.346 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2024-03-07T13:43:43.437 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:45:43.599 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:47:43.768 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:48:15.846 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:48:55.107 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:49:56.496 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 504: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:50:58.064 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 504: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:50:59.820 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:01.762 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:03.890 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:06.206 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:08.748 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:11.438 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:11.439 ERROR:teuthology.dispatcher.supervisor:Failed to report job status; ignoring
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 335, in run_with_watchdog
    report.try_push_job_info(job_info)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/report.py", line 512, in try_push_job_info
    push_job_info(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/report.py", line 474, in push_job_info
    reporter.report_job(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/report.py", line 299, in report_job
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'report job 7584479' reached maximum tries (11) after waiting for 18.4301756803593 seconds
2024-03-07T13:53:11.528 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:53:11.559 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:53:12.647 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:53:13.818 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:53:15.072 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:53:16.410 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:53:17.830 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:53:19.335 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:53:20.922 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:53:22.594 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T19:09:52.953 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:09:53.031 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1