2023-11-04T01:05:32.440 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-04T01:05:32.441 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:05:32.585 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f28caafba60>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f28ca83bca0>, signals=[15])
2023-11-04T01:05:32.585 INFO:teuthology.lock.ops:Start node 'smithi174.front.sepia.ceph.com' reimaging
2023-11-04T01:05:32.585 INFO:teuthology.lock.ops:Updating [smithi174.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T01:05:32.585 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server
2023-11-04T01:05:32.601 INFO:teuthology.lock.ops:Start node 'smithi177.front.sepia.ceph.com' reimaging
2023-11-04T01:05:32.602 INFO:teuthology.lock.ops:Updating [smithi177.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T01:05:32.602 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server
2023-11-04T01:05:32.603 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /host {"name": "smithi174"}
2023-11-04T01:05:32.627 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /host {"name": "smithi177"}
2023-11-04T01:05:32.666 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /image {"name": "smithi_centos_9.stream"}
2023-11-04T01:05:32.702 DEBUG:teuthology.provision.fog.smithi174:Requesting image smithi_centos_9.stream (ID 10)
2023-11-04T01:05:32.702 DEBUG:teuthology.provision.fog.smithi174:FOG request: PUT /host/279 {"imageID": 10}
2023-11-04T01:05:32.705 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /image {"name": "smithi_centos_9.stream"}
2023-11-04T01:05:32.753 DEBUG:teuthology.provision.fog.smithi177:Requesting image smithi_centos_9.stream (ID 10)
2023-11-04T01:05:32.753 DEBUG:teuthology.provision.fog.smithi177:FOG request: PUT /host/282 {"imageID": 10}
2023-11-04T01:05:32.757 INFO:teuthology.provision.fog.smithi174:Scheduling deploy of centos 9.stream
2023-11-04T01:05:32.757 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:05:32.809 INFO:teuthology.provision.fog.smithi177:Scheduling deploy of centos 9.stream
2023-11-04T01:05:32.809 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:05:32.913 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T01:05:32.952 DEBUG:teuthology.provision.fog.smithi174:FOG request: POST /host/279/task {"taskTypeID": 1}
2023-11-04T01:05:33.006 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T01:05:33.009 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:05:33.040 DEBUG:teuthology.provision.fog.smithi177:FOG request: POST /host/282/task {"taskTypeID": 1}
2023-11-04T01:05:33.107 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:05:33.239 INFO:teuthology.orchestra.console.smithi174:Power off
2023-11-04T01:05:33.239 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T01:05:33.260 DEBUG:teuthology.orchestra.console.smithi174:power off output: Chassis Power Control: Down/Off
2023-11-04T01:05:33.260 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T01:05:33.282 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on
2023-11-04T01:05:33.345 INFO:teuthology.orchestra.console.smithi177:Power off
2023-11-04T01:05:33.345 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T01:05:33.366 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off
2023-11-04T01:05:33.366 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T01:05:33.388 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on
2023-11-04T01:05:37.287 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T01:05:37.390 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T01:05:37.398 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on
2023-11-04T01:05:37.498 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on
2023-11-04T01:05:41.402 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T01:05:41.499 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T01:05:41.511 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off
2023-11-04T01:05:41.608 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off
2023-11-04T01:05:41.612 INFO:teuthology.orchestra.console.smithi174:Power off completed
2023-11-04T01:05:41.709 INFO:teuthology.orchestra.console.smithi177:Power off completed
2023-11-04T01:05:41.714 INFO:teuthology.orchestra.console.smithi174:Power on
2023-11-04T01:05:41.714 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T01:05:41.733 DEBUG:teuthology.orchestra.console.smithi174:power on output: Chassis Power Control: Up/On
2023-11-04T01:05:41.734 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T01:05:41.753 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off
2023-11-04T01:05:41.810 INFO:teuthology.orchestra.console.smithi177:Power on
2023-11-04T01:05:41.810 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T01:05:41.830 DEBUG:teuthology.orchestra.console.smithi177:power on output: Chassis Power Control: Up/On
2023-11-04T01:05:41.830 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T01:05:41.850 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off
2023-11-04T01:05:45.757 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T01:05:45.851 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T01:05:45.865 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off
2023-11-04T01:05:45.959 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off
2023-11-04T01:05:49.869 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T01:05:49.960 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T01:05:49.978 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on
2023-11-04T01:05:50.068 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on
2023-11-04T01:05:50.080 INFO:teuthology.orchestra.console.smithi174:Power on completed
2023-11-04T01:05:50.169 INFO:teuthology.orchestra.console.smithi177:Power on completed
2023-11-04T01:05:50.181 INFO:teuthology.provision.fog.smithi174:Waiting for deploy to finish
2023-11-04T01:05:50.181 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:05:50.270 INFO:teuthology.provision.fog.smithi177:Waiting for deploy to finish
2023-11-04T01:05:50.270 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:06:05.418 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:06:05.487 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:06:20.716 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:06:20.734 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:06:35.981 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:06:35.984 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:07:16.869 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:07:17.613 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:07:34.311 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:07:38.986 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:07:52.634 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:08:01.142 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:08:08.661 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:08:22.066 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:08:25.345 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:08:40.770 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:08:43.976 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:09:00.414 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:09:03.398 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:09:19.379 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:09:21.722 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:09:36.203 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:09:38.220 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:09:53.534 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:09:55.582 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:10:09.830 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:10:12.553 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:10:26.026 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:10:28.984 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:10:41.931 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-11-04T01:10:42.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T01:10:43.227 ERROR:teuthology.orchestra.connection:Error authenticating with smithi174.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2023-11-04T01:10:44.965 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:11:00.454 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:11:16.134 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:11:31.379 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:11:46.561 DEBUG:teuthology.provision.fog.smithi177:FOG request: GET /task/active None
2023-11-04T01:22:49.662 WARNING:teuthology.provision.fog.smithi174:[Errno 110] Connection timed out
2023-11-04T01:22:50.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T01:22:50.450 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-11-04T01:22:50.451 DEBUG:teuthology.orchestra.run.smithi177:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-11-04T01:22:50.764 INFO:teuthology.provision.fog.smithi177:Node is ready
2023-11-04T01:22:50.764 DEBUG:teuthology.orchestra.run.smithi177:> hostname
2023-11-04T01:22:50.778 INFO:teuthology.orchestra.run.smithi177.stdout:smithi177.front.sepia.ceph.com
2023-11-04T01:22:50.778 DEBUG:teuthology.orchestra.run.smithi177:> grep smithi177.front.sepia.ceph.com /etc/hosts
2023-11-04T01:22:50.833 INFO:teuthology.orchestra.run.smithi177.stdout:172.21.15.177 smithi177.front.sepia.ceph.com smithi177
2023-11-04T01:22:50.834 DEBUG:teuthology.orchestra.run.smithi177:> sudo hostname smithi177
2023-11-04T01:22:50.950 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i -e 's/smithi177.front.sepia.ceph.com/smithi177/g' /etc/hosts
2023-11-04T01:22:50.977 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i -e 's/172.21.15.177/172.21.15.177/g' /etc/hosts
2023-11-04T01:22:51.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i -e 's/smithi177.front.sepia.ceph.com/smithi177/g' /etc/hostname
2023-11-04T01:22:51.111 DEBUG:teuthology.orchestra.run.smithi177:> sudo hostname smithi177
2023-11-04T01:22:51.178 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release
2023-11-04T01:22:51.233 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream"
2023-11-04T01:22:51.233 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9"
2023-11-04T01:22:51.233 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos"
2023-11-04T01:22:51.233 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora"
2023-11-04T01:22:51.233 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9"
2023-11-04T01:22:51.233 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9"
2023-11-04T01:22:51.234 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9"
2023-11-04T01:22:51.234 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31"
2023-11-04T01:22:51.234 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon"
2023-11-04T01:22:51.234 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2023-11-04T01:22:51.234 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/"
2023-11-04T01:22:51.234 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-11-04T01:22:51.234 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2023-11-04T01:22:51.234 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-11-04T01:22:51.235 INFO:teuthology.provision.fog.smithi177:Deploy complete!
2023-11-04T01:22:51.236 DEBUG:teuthology.parallel:result is None
2023-11-04T01:22:55.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T01:22:55.887 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-11-04T01:22:55.887 DEBUG:teuthology.orchestra.run.smithi174:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-11-04T01:22:56.235 INFO:teuthology.provision.fog.smithi174:Node is ready
2023-11-04T01:22:56.235 DEBUG:teuthology.orchestra.run.smithi174:> hostname
2023-11-04T01:22:56.251 INFO:teuthology.orchestra.run.smithi174.stdout:smithi174.front.sepia.ceph.com
2023-11-04T01:22:56.251 DEBUG:teuthology.orchestra.run.smithi174:> grep smithi174.front.sepia.ceph.com /etc/hosts
2023-11-04T01:22:56.307 INFO:teuthology.orchestra.run.smithi174.stdout:172.21.15.174 smithi174.front.sepia.ceph.com smithi174
2023-11-04T01:22:56.307 DEBUG:teuthology.orchestra.run.smithi174:> sudo hostname smithi174
2023-11-04T01:22:56.417 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i -e 's/smithi174.front.sepia.ceph.com/smithi174/g' /etc/hosts
2023-11-04T01:22:56.447 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i -e 's/172.21.15.174/172.21.15.174/g' /etc/hosts
2023-11-04T01:22:56.515 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i -e 's/smithi174.front.sepia.ceph.com/smithi174/g' /etc/hostname
2023-11-04T01:22:56.583 DEBUG:teuthology.orchestra.run.smithi174:> sudo hostname smithi174
2023-11-04T01:22:56.650 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release
2023-11-04T01:22:56.704 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream"
2023-11-04T01:22:56.705 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9"
2023-11-04T01:22:56.705 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos"
2023-11-04T01:22:56.705 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora"
2023-11-04T01:22:56.705 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9"
2023-11-04T01:22:56.705 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9"
2023-11-04T01:22:56.705 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9"
2023-11-04T01:22:56.705 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31"
2023-11-04T01:22:56.705 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon"
2023-11-04T01:22:56.705 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2023-11-04T01:22:56.705 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/"
2023-11-04T01:22:56.705 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-11-04T01:22:56.705 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2023-11-04T01:22:56.705 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-11-04T01:22:56.706 INFO:teuthology.provision.fog.smithi174:Deploy complete!
2023-11-04T01:22:56.706 DEBUG:teuthology.parallel:result is None
2023-11-04T01:22:56.870 INFO:teuthology.lock.ops:Checking smithi174.front.sepia.ceph.com
2023-11-04T01:22:56.871 INFO:teuthology.lock.ops:New key found. Updating...
2023-11-04T01:22:56.891 INFO:teuthology.lock.ops:Checking smithi177.front.sepia.ceph.com
2023-11-04T01:22:56.891 INFO:teuthology.lock.ops:New key found. Updating...
2023-11-04T01:22:56.909 INFO:teuthology.lock.ops:Updating [smithi174.front.sepia.ceph.com]: set os type and version on server
2023-11-04T01:22:56.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com'
2023-11-04T01:22:56.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T01:22:56.991 DEBUG:teuthology.orchestra.run.smithi174:> true
2023-11-04T01:22:57.060 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com'
2023-11-04T01:22:57.060 DEBUG:teuthology.orchestra.run.smithi174:> uname -m
2023-11-04T01:22:57.114 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64
2023-11-04T01:22:57.114 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release
2023-11-04T01:22:57.168 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream"
2023-11-04T01:22:57.169 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9"
2023-11-04T01:22:57.169 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos"
2023-11-04T01:22:57.169 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora"
2023-11-04T01:22:57.169 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9"
2023-11-04T01:22:57.169 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9"
2023-11-04T01:22:57.169 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9"
2023-11-04T01:22:57.169 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31"
2023-11-04T01:22:57.169 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon"
2023-11-04T01:22:57.169 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2023-11-04T01:22:57.169 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/"
2023-11-04T01:22:57.169 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-11-04T01:22:57.169 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2023-11-04T01:22:57.170 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-11-04T01:22:57.170 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server
2023-11-04T01:22:57.238 INFO:teuthology.lock.ops:Updating [smithi177.front.sepia.ceph.com]: set os type and version on server
2023-11-04T01:22:57.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com'
2023-11-04T01:22:57.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T01:22:57.323 DEBUG:teuthology.orchestra.run.smithi177:> true
2023-11-04T01:22:57.394 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com'
2023-11-04T01:22:57.394 DEBUG:teuthology.orchestra.run.smithi177:> uname -m
2023-11-04T01:22:57.451 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64
2023-11-04T01:22:57.451 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release
2023-11-04T01:22:57.508 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream"
2023-11-04T01:22:57.508 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9"
2023-11-04T01:22:57.508 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos"
2023-11-04T01:22:57.508 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora"
2023-11-04T01:22:57.508 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9"
2023-11-04T01:22:57.508 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9"
2023-11-04T01:22:57.508 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9"
2023-11-04T01:22:57.508 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31"
2023-11-04T01:22:57.509 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon"
2023-11-04T01:22:57.509 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2023-11-04T01:22:57.509 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/"
2023-11-04T01:22:57.509 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-11-04T01:22:57.509 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2023-11-04T01:22:57.509 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-11-04T01:22:57.509 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server
2023-11-04T01:22:57.532 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:22:57.627 INFO:teuthology.dispatcher.supervisor:Running job 7445941
2023-11-04T01:22:57.627 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology -v --owner scheduled_cbodley@teuthology --archive /home/teuthworker/archive/cbodley-2023-11-03_18:34:54-rgw-wip-cbodley-testing-distro-default-smithi/7445941 --name cbodley-2023-11-03_18:34:54-rgw-wip-cbodley-testing-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi} -- /home/teuthworker/archive/cbodley-2023-11-03_18:34:54-rgw-wip-cbodley-testing-distro-default-smithi/7445941/orig.config.yaml
2023-11-04T01:22:57.630 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/cbodley-2023-11-03_18:34:54-rgw-wip-cbodley-testing-distro-default-smithi/7445941
2023-11-04T01:22:57.630 INFO:teuthology.dispatcher.supervisor:Job PID: 3098551
2023-11-04T01:22:57.630 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-11-04T01:24:57.631 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:26:57.710 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:28:57.783 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:30:57.865 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:32:57.985 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:34:58.074 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:36:58.168 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:38:58.238 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:40:58.311 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:42:58.370 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:44:58.424 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:46:58.484 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:48:58.557 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:50:58.624 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:52:58.674 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:54:58.740 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:56:58.819 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T01:58:58.888 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:00:58.961 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:02:59.024 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:04:59.092 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:06:59.161 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:08:59.225 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:10:59.298 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:12:59.366 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:14:59.448 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:16:59.525 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:18:59.599 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:20:59.672 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:22:59.740 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:24:59.814 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:26:59.879 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:28:59.947 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:31:00.004 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:33:00.066 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:35:00.124 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:37:00.187 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:39:00.254 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:41:00.328 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:43:00.395 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:45:00.472 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:47:00.546 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:49:00.619 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:51:00.697 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:53:00.763 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:55:00.830 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:57:00.899 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T02:59:00.973 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:01:01.069 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:03:01.138 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:05:01.195 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:07:01.255 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:09:01.316 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:11:01.391 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:13:01.460 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:15:01.539 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:17:01.608 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:19:01.684 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:21:01.751 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:23:01.820 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:25:01.885 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:27:01.963 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:29:02.022 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:31:02.089 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:33:02.152 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:35:02.224 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:37:02.285 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:39:02.353 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:41:02.417 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:43:02.484 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:45:02.557 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:47:02.645 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:49:02.707 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:49:02.773 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120
2023-11-04T03:49:02.960 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-11-04T03:49:02.960 INFO:teuthology.nuke:Checking targets against current locks
2023-11-04T03:49:03.010 DEBUG:teuthology.nuke:shortname: smithi174
2023-11-04T03:49:03.010 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T03:49:03.032 DEBUG:teuthology.nuke:shortname: smithi177
2023-11-04T03:49:03.032 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T03:49:03.036 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cbodley-2023-11-03_18:34:54-rgw-wip-cbodley-testing-distro-default-smithi/7445941', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 01:05:31.644247', 'locked_by': 'scheduled_cbodley@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK0dRe2nVf3XGe+1/IOw2kFQl2V0TWTYfozaM2r4clyYX07M2+xntdOoxPmwgS+YjiO9hZHC88nCPxAH1n0MUKk='}
2023-11-04T03:49:03.058 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cbodley-2023-11-03_18:34:54-rgw-wip-cbodley-testing-distro-default-smithi/7445941', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 01:05:31.645682', 'locked_by': 'scheduled_cbodley@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLhPEgNzdxinMFVqgt72qJzGwdMw7Q6CSsyvaubMeqhXudq42WMiuuEoYt7lBOVzPQtrVxG0Mw6eLImDsxMX+d8='}
2023-11-04T03:49:03.087 INFO:teuthology.orchestra.console.smithi174:Power off
2023-11-04T03:49:03.087 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:49:03.110 DEBUG:teuthology.orchestra.console.smithi174:power off output: Chassis Power Control: Down/Off
2023-11-04T03:49:03.110 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:49:03.125 INFO:teuthology.orchestra.console.smithi177:Power off
2023-11-04T03:49:03.125 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:49:03.133 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off
2023-11-04T03:49:03.146 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off
2023-11-04T03:49:03.146 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:49:03.170 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off
2023-11-04T03:49:03.234 INFO:teuthology.orchestra.console.smithi174:Power off completed
2023-11-04T03:49:03.271 INFO:teuthology.orchestra.console.smithi177:Power off completed
2023-11-04T03:49:03.388 INFO:teuthology.lock.ops:unlocked: smithi174.front.sepia.ceph.com
2023-11-04T03:49:03.421 INFO:teuthology.lock.ops:unlocked: smithi177.front.sepia.ceph.com