2024-03-07T13:37:36.557 INFO:root:teuthology version: 0.0.1.dev262+ge691533
2024-03-07T13:37:36.558 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:37:36.706 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fe8d066af40>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fe8d16cbc10>, signals=[15])
2024-03-07T13:37:36.706 INFO:teuthology.lock.ops:Start node 'smithi115.front.sepia.ceph.com' reimaging
2024-03-07T13:37:36.707 INFO:teuthology.lock.ops:Updating [smithi115.front.sepia.ceph.com]: reset os type and version on server
2024-03-07T13:37:36.707 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server
2024-03-07T13:37:36.725 INFO:teuthology.lock.ops:Start node 'smithi129.front.sepia.ceph.com' reimaging
2024-03-07T13:37:36.726 INFO:teuthology.lock.ops:Updating [smithi129.front.sepia.ceph.com]: reset os type and version on server
2024-03-07T13:37:36.726 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server
2024-03-07T13:37:36.841 DEBUG:teuthology.provision.fog.smithi115:Requesting image smithi_centos_9.stream (ID 10)
2024-03-07T13:37:36.865 DEBUG:teuthology.provision.fog.smithi129:Requesting image smithi_centos_9.stream (ID 10)
2024-03-07T13:37:36.887 INFO:teuthology.provision.fog.smithi115:Scheduling deploy of centos 9.stream
2024-03-07T13:37:36.905 INFO:teuthology.provision.fog.smithi129:Scheduling deploy of centos 9.stream
2024-03-07T13:37:37.188 INFO:teuthology.orchestra.console.smithi115:Power off
2024-03-07T13:37:37.188 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-07T13:37:37.207 INFO:teuthology.orchestra.console.smithi129:Power off
2024-03-07T13:37:37.207 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-07T13:37:37.216 DEBUG:teuthology.orchestra.console.smithi115:power off output: Chassis Power Control: Down/Off
2024-03-07T13:37:37.216 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:37.229 DEBUG:teuthology.orchestra.console.smithi129:power off output: Chassis Power Control: Down/Off
2024-03-07T13:37:37.230 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:37.238 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is off
2024-03-07T13:37:37.251 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is off
2024-03-07T13:37:37.339 INFO:teuthology.orchestra.console.smithi115:Power off completed
2024-03-07T13:37:37.352 INFO:teuthology.orchestra.console.smithi129:Power off completed
2024-03-07T13:37:37.440 INFO:teuthology.orchestra.console.smithi115:Power on
2024-03-07T13:37:37.441 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-07T13:37:37.453 INFO:teuthology.orchestra.console.smithi129:Power on
2024-03-07T13:37:37.453 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-07T13:37:37.464 DEBUG:teuthology.orchestra.console.smithi115:power on output: Chassis Power Control: Up/On
2024-03-07T13:37:37.464 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:37.477 DEBUG:teuthology.orchestra.console.smithi129:power on output: Chassis Power Control: Up/On
2024-03-07T13:37:37.478 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:37.493 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is off
2024-03-07T13:37:37.499 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is off
2024-03-07T13:37:41.496 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:41.503 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:41.605 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on
2024-03-07T13:37:41.612 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is off
2024-03-07T13:37:41.707 INFO:teuthology.orchestra.console.smithi115:Power on completed
2024-03-07T13:37:41.808 INFO:teuthology.provision.fog.smithi115:Waiting for deploy to finish
2024-03-07T13:37:45.616 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:45.801 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on
2024-03-07T13:37:45.901 INFO:teuthology.orchestra.console.smithi129:Power on completed
2024-03-07T13:37:46.002 INFO:teuthology.provision.fog.smithi129:Waiting for deploy to finish
2024-03-07T13:40:17.133 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:20.188 WARNING:teuthology.provision.fog.smithi129:[Errno None] Unable to connect to port 22 on 172.21.15.129
2024-03-07T13:40:26.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:28.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:29.244 WARNING:teuthology.provision.fog.smithi129:[Errno None] Unable to connect to port 22 on 172.21.15.129
2024-03-07T13:40:31.068 WARNING:teuthology.provision.fog.smithi115:[Errno None] Unable to connect to port 22 on 172.21.15.115
2024-03-07T13:40:35.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:37.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:38.300 WARNING:teuthology.provision.fog.smithi129:[Errno None] Unable to connect to port 22 on 172.21.15.129
2024-03-07T13:40:40.124 WARNING:teuthology.provision.fog.smithi115:[Errno None] Unable to connect to port 22 on 172.21.15.115
2024-03-07T13:40:44.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:46.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:47.360 WARNING:teuthology.provision.fog.smithi129:[Errno None] Unable to connect to port 22 on 172.21.15.129
2024-03-07T13:40:49.180 WARNING:teuthology.provision.fog.smithi115:[Errno None] Unable to connect to port 22 on 172.21.15.115
2024-03-07T13:40:53.365 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:55.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:56.444 WARNING:teuthology.provision.fog.smithi129:[Errno None] Unable to connect to port 22 on 172.21.15.129
2024-03-07T13:40:58.236 WARNING:teuthology.provision.fog.smithi115:[Errno None] Unable to connect to port 22 on 172.21.15.115
2024-03-07T13:41:02.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:04.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:05.500 WARNING:teuthology.provision.fog.smithi129:[Errno None] Unable to connect to port 22 on 172.21.15.129
2024-03-07T13:41:07.292 WARNING:teuthology.provision.fog.smithi115:[Errno None] Unable to connect to port 22 on 172.21.15.115
2024-03-07T13:41:11.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:13.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:16.348 WARNING:teuthology.provision.fog.smithi115:[Errno None] Unable to connect to port 22 on 172.21.15.115
2024-03-07T13:41:22.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:42:11.553 WARNING:teuthology.provision.fog.smithi129:timed out
2024-03-07T13:42:17.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:42:17.789 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-07T13:42:17.789 DEBUG:teuthology.orchestra.run.smithi129:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-07T13:42:18.093 INFO:teuthology.provision.fog.smithi129:Node is ready
2024-03-07T13:42:18.093 DEBUG:teuthology.orchestra.run.smithi129:> hostname
2024-03-07T13:42:18.108 INFO:teuthology.orchestra.run.smithi129.stdout:smithi129.front.sepia.ceph.com
2024-03-07T13:42:18.109 DEBUG:teuthology.orchestra.run.smithi129:> grep smithi129.front.sepia.ceph.com /etc/hosts
2024-03-07T13:42:18.164 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129 smithi129.front.sepia.ceph.com smithi129
2024-03-07T13:42:18.164 DEBUG:teuthology.orchestra.run.smithi129:> sudo hostname smithi129
2024-03-07T13:42:18.279 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i -e 's/smithi129.front.sepia.ceph.com/smithi129/g' /etc/hosts
2024-03-07T13:42:18.309 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i -e 's/172.21.15.129/172.21.15.129/g' /etc/hosts
2024-03-07T13:42:18.377 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i -e 's/smithi129.front.sepia.ceph.com/smithi129/g' /etc/hostname
2024-03-07T13:42:18.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo hostname smithi129
2024-03-07T13:42:18.513 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release
2024-03-07T13:42:18.567 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream"
2024-03-07T13:42:18.567 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9"
2024-03-07T13:42:18.567 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos"
2024-03-07T13:42:18.568 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora"
2024-03-07T13:42:18.568 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9"
2024-03-07T13:42:18.568 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9"
2024-03-07T13:42:18.568 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-07T13:42:18.568 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31"
2024-03-07T13:42:18.568 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon"
2024-03-07T13:42:18.568 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-07T13:42:18.568 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/"
2024-03-07T13:42:18.568 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-07T13:42:18.568 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-07T13:42:18.568 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-07T13:42:18.569 INFO:teuthology.provision.fog.smithi129:Deploy complete!
2024-03-07T13:42:18.571 DEBUG:teuthology.parallel:result is None
2024-03-07T13:42:22.362 WARNING:teuthology.provision.fog.smithi115:timed out
2024-03-07T13:42:28.369 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:42:28.559 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-07T13:42:28.559 DEBUG:teuthology.orchestra.run.smithi115:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-07T13:42:28.867 INFO:teuthology.provision.fog.smithi115:Node is ready
2024-03-07T13:42:28.867 DEBUG:teuthology.orchestra.run.smithi115:> hostname
2024-03-07T13:42:28.881 INFO:teuthology.orchestra.run.smithi115.stdout:smithi115.front.sepia.ceph.com
2024-03-07T13:42:28.881 DEBUG:teuthology.orchestra.run.smithi115:> grep smithi115.front.sepia.ceph.com /etc/hosts
2024-03-07T13:42:28.936 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115 smithi115.front.sepia.ceph.com smithi115
2024-03-07T13:42:28.937 DEBUG:teuthology.orchestra.run.smithi115:> sudo hostname smithi115
2024-03-07T13:42:29.052 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i -e 's/smithi115.front.sepia.ceph.com/smithi115/g' /etc/hosts
2024-03-07T13:42:29.079 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i -e 's/172.21.15.115/172.21.15.115/g' /etc/hosts
2024-03-07T13:42:29.146 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i -e 's/smithi115.front.sepia.ceph.com/smithi115/g' /etc/hostname
2024-03-07T13:42:29.213 DEBUG:teuthology.orchestra.run.smithi115:> sudo hostname smithi115
2024-03-07T13:42:29.279 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release
2024-03-07T13:42:29.333 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="CentOS Stream"
2024-03-07T13:42:29.333 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="9"
2024-03-07T13:42:29.333 INFO:teuthology.orchestra.run.smithi115.stdout:ID="centos"
2024-03-07T13:42:29.333 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="rhel fedora"
2024-03-07T13:42:29.334 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="9"
2024-03-07T13:42:29.334 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el9"
2024-03-07T13:42:29.334 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-07T13:42:29.334 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31"
2024-03-07T13:42:29.334 INFO:teuthology.orchestra.run.smithi115.stdout:LOGO="fedora-logo-icon"
2024-03-07T13:42:29.334 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-07T13:42:29.334 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://centos.org/"
2024-03-07T13:42:29.334 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-07T13:42:29.334 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-07T13:42:29.334 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-07T13:42:29.334 INFO:teuthology.provision.fog.smithi115:Deploy complete!
2024-03-07T13:42:29.335 DEBUG:teuthology.parallel:result is None
2024-03-07T13:42:29.499 INFO:teuthology.lock.ops:Checking smithi115.front.sepia.ceph.com
2024-03-07T13:42:29.499 INFO:teuthology.lock.ops:New key found. Updating...
2024-03-07T13:42:29.525 INFO:teuthology.lock.ops:Checking smithi129.front.sepia.ceph.com
2024-03-07T13:42:29.525 INFO:teuthology.lock.ops:New key found. Updating...
2024-03-07T13:42:29.543 INFO:teuthology.lock.ops:Updating [smithi115.front.sepia.ceph.com]: set os type and version on server
2024-03-07T13:42:29.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com'
2024-03-07T13:42:29.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:42:29.633 DEBUG:teuthology.orchestra.run.smithi115:> true
2024-03-07T13:42:29.698 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com'
2024-03-07T13:42:29.698 DEBUG:teuthology.orchestra.run.smithi115:> uname -m
2024-03-07T13:42:29.753 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64
2024-03-07T13:42:29.753 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release
2024-03-07T13:42:29.808 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="CentOS Stream"
2024-03-07T13:42:29.808 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="9"
2024-03-07T13:42:29.808 INFO:teuthology.orchestra.run.smithi115.stdout:ID="centos"
2024-03-07T13:42:29.808 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="rhel fedora"
2024-03-07T13:42:29.808 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="9"
2024-03-07T13:42:29.808 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el9"
2024-03-07T13:42:29.808 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-07T13:42:29.808 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31"
2024-03-07T13:42:29.808 INFO:teuthology.orchestra.run.smithi115.stdout:LOGO="fedora-logo-icon"
2024-03-07T13:42:29.808 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-07T13:42:29.809 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://centos.org/"
2024-03-07T13:42:29.809 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-07T13:42:29.809 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-07T13:42:29.809 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-07T13:42:29.809 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server
2024-03-07T13:42:29.834 INFO:teuthology.lock.ops:Updating [smithi129.front.sepia.ceph.com]: set os type and version on server
2024-03-07T13:42:29.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com'
2024-03-07T13:42:29.835 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:42:29.922 DEBUG:teuthology.orchestra.run.smithi129:> true
2024-03-07T13:42:30.089 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com'
2024-03-07T13:42:30.090 DEBUG:teuthology.orchestra.run.smithi129:> uname -m
2024-03-07T13:42:30.144 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64
2024-03-07T13:42:30.145 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release
2024-03-07T13:42:30.199 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream"
2024-03-07T13:42:30.199 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9"
2024-03-07T13:42:30.199 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos"
2024-03-07T13:42:30.199 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora"
2024-03-07T13:42:30.199 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9"
2024-03-07T13:42:30.199 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9"
2024-03-07T13:42:30.200 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-07T13:42:30.200 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31"
2024-03-07T13:42:30.200 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon"
2024-03-07T13:42:30.200 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-07T13:42:30.200 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/"
2024-03-07T13:42:30.200 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-07T13:42:30.200 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-07T13:42:30.200 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-07T13:42:30.200 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server
2024-03-07T13:42:30.225 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:42:30.317 INFO:teuthology.dispatcher.supervisor:Running job 7584487
2024-03-07T13:42:30.317 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7584487 --name vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi --description fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} -- /home/teuthworker/archive/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7584487/orig.config.yaml
2024-03-07T13:42:30.319 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7584487
2024-03-07T13:42:30.320 INFO:teuthology.dispatcher.supervisor:Job PID: 692556
2024-03-07T13:42:30.320 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2024-03-07T13:44:30.421 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:46:30.586 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:48:30.758 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:49:30.792 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 504: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:50:32.412 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 504: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:50:59.609 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:02.457 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:05.919 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:09.996 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:14.686 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:19.993 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:25.910 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:32.441 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:32.441 ERROR:teuthology.dispatcher.supervisor:Failed to report job status; ignoring
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 335, in run_with_watchdog
    report.try_push_job_info(job_info)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/report.py", line 512, in try_push_job_info
    push_job_info(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/report.py", line 474, in push_job_info
    reporter.report_job(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/report.py", line 299, in report_job
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'report job 7584487' reached maximum tries (11) after waiting for 37.62433250547069 seconds
2024-03-07T13:53:32.540 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:53:32.675 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1