2023-08-24T09:46:00.051 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc
2023-08-24T09:46:00.053 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-24T09:46:00.152 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f78154f1850>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f7814e12e50>, signals=[15])
2023-08-24T09:46:00.154 INFO:teuthology.lock.ops:Start node 'smithi044.front.sepia.ceph.com' reimaging
2023-08-24T09:46:00.154 INFO:teuthology.lock.ops:Updating [smithi044.front.sepia.ceph.com]: reset os type and version on server
2023-08-24T09:46:00.154 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server
2023-08-24T09:46:00.185 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /host {"name": "smithi044"}
2023-08-24T09:46:00.258 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /image {"name": "smithi_centos_9.stream"}
2023-08-24T09:46:00.301 DEBUG:teuthology.provision.fog.smithi044:Requesting image smithi_centos_9.stream (ID 10)
2023-08-24T09:46:00.302 DEBUG:teuthology.provision.fog.smithi044:FOG request: PUT /host/149 {"imageID": 10}
2023-08-24T09:46:00.349 INFO:teuthology.provision.fog.smithi044:Scheduling deploy of centos 9.stream
2023-08-24T09:46:00.350 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /task/active None
2023-08-24T09:46:00.411 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /tasktype {"name": "deploy"}
2023-08-24T09:46:00.456 DEBUG:teuthology.provision.fog.smithi044:FOG request: POST /host/149/task {"taskTypeID": 1}
2023-08-24T09:46:00.522 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /task/active None
2023-08-24T09:46:00.618 INFO:teuthology.orchestra.console.smithi044:Power off
2023-08-24T09:46:00.619 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-24T09:46:00.642 DEBUG:teuthology.orchestra.console.smithi044:power off output: Chassis Power Control: Down/Off
2023-08-24T09:46:00.643 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-24T09:46:00.668 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on
2023-08-24T09:46:04.673 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-24T09:46:04.783 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on
2023-08-24T09:46:08.784 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-24T09:46:08.894 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is off
2023-08-24T09:46:08.995 INFO:teuthology.orchestra.console.smithi044:Power off completed
2023-08-24T09:46:09.096 INFO:teuthology.orchestra.console.smithi044:Power on
2023-08-24T09:46:09.096 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-08-24T09:46:09.120 DEBUG:teuthology.orchestra.console.smithi044:power on output: Chassis Power Control: Up/On
2023-08-24T09:46:09.121 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-24T09:46:09.146 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is off
2023-08-24T09:46:13.149 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-24T09:46:13.260 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is off
2023-08-24T09:46:17.264 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-24T09:46:17.374 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on
2023-08-24T09:46:17.475 INFO:teuthology.orchestra.console.smithi044:Power on completed
2023-08-24T09:46:17.577 INFO:teuthology.provision.fog.smithi044:Waiting for deploy to finish
2023-08-24T09:46:17.577 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /task/active None
2023-08-24T09:46:32.662 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /task/active None
2023-08-24T09:46:47.752 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /task/active None
2023-08-24T09:47:02.836 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /task/active None
2023-08-24T09:47:17.928 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /task/active None
2023-08-24T09:47:33.006 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /task/active None
2023-08-24T09:47:48.091 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /task/active None
2023-08-24T09:48:03.183 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /task/active None
2023-08-24T09:48:18.272 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /task/active None
2023-08-24T09:48:33.372 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /task/active None
2023-08-24T09:48:48.460 DEBUG:teuthology.provision.fog.smithi044:FOG request: GET /task/active None
2023-08-24T09:48:48.540 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-24T09:48:51.604 WARNING:teuthology.provision.fog.smithi044:[Errno None] Unable to connect to port 22 on 172.21.15.44
2023-08-24T09:48:57.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-24T09:49:00.688 WARNING:teuthology.provision.fog.smithi044:[Errno None] Unable to connect to port 22 on 172.21.15.44
2023-08-24T09:49:06.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-24T09:49:09.744 WARNING:teuthology.provision.fog.smithi044:[Errno None] Unable to connect to port 22 on 172.21.15.44
2023-08-24T09:49:15.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-24T09:49:18.832 WARNING:teuthology.provision.fog.smithi044:[Errno None] Unable to connect to port 22 on 172.21.15.44
2023-08-24T09:49:24.836 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-24T09:49:27.888 WARNING:teuthology.provision.fog.smithi044:[Errno None] Unable to connect to port 22 on 172.21.15.44
2023-08-24T09:49:33.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-24T09:50:33.952 WARNING:teuthology.provision.fog.smithi044:timed out
2023-08-24T09:50:39.956 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-24T09:50:40.170 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-08-24T09:50:40.171 DEBUG:teuthology.orchestra.run.smithi044:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-08-24T09:50:40.470 INFO:teuthology.provision.fog.smithi044:Node is ready
2023-08-24T09:50:40.470 DEBUG:teuthology.orchestra.run.smithi044:> hostname
2023-08-24T09:50:40.486 INFO:teuthology.orchestra.run.smithi044.stdout:smithi044.front.sepia.ceph.com
2023-08-24T09:50:40.486 DEBUG:teuthology.orchestra.run.smithi044:> grep smithi044.front.sepia.ceph.com /etc/hosts
2023-08-24T09:50:40.543 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44 smithi044.front.sepia.ceph.com smithi044
2023-08-24T09:50:40.543 DEBUG:teuthology.orchestra.run.smithi044:> sudo hostname smithi044
2023-08-24T09:50:40.666 DEBUG:teuthology.orchestra.run.smithi044:> sudo sed -i -e 's/smithi044.front.sepia.ceph.com/smithi044/g' /etc/hosts
2023-08-24T09:50:40.699 DEBUG:teuthology.orchestra.run.smithi044:> sudo sed -i -e 's/172.21.15.44/172.21.15.44/g' /etc/hosts
2023-08-24T09:50:40.770 DEBUG:teuthology.orchestra.run.smithi044:> sudo sed -i -e 's/smithi044.front.sepia.ceph.com/smithi044/g' /etc/hostname
2023-08-24T09:50:40.842 DEBUG:teuthology.orchestra.run.smithi044:> sudo hostname smithi044
2023-08-24T09:50:40.913 DEBUG:teuthology.orchestra.run.smithi044:> cat /etc/os-release
2023-08-24T09:50:40.972 INFO:teuthology.orchestra.run.smithi044.stdout:NAME="CentOS Stream"
2023-08-24T09:50:40.973 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION="9"
2023-08-24T09:50:40.973 INFO:teuthology.orchestra.run.smithi044.stdout:ID="centos"
2023-08-24T09:50:40.973 INFO:teuthology.orchestra.run.smithi044.stdout:ID_LIKE="rhel fedora"
2023-08-24T09:50:40.973 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_ID="9"
2023-08-24T09:50:40.973 INFO:teuthology.orchestra.run.smithi044.stdout:PLATFORM_ID="platform:el9"
2023-08-24T09:50:40.973 INFO:teuthology.orchestra.run.smithi044.stdout:PRETTY_NAME="CentOS Stream 9"
2023-08-24T09:50:40.974 INFO:teuthology.orchestra.run.smithi044.stdout:ANSI_COLOR="0;31"
2023-08-24T09:50:40.974 INFO:teuthology.orchestra.run.smithi044.stdout:LOGO="fedora-logo-icon"
2023-08-24T09:50:40.974 INFO:teuthology.orchestra.run.smithi044.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2023-08-24T09:50:40.974 INFO:teuthology.orchestra.run.smithi044.stdout:HOME_URL="https://centos.org/"
2023-08-24T09:50:40.974 INFO:teuthology.orchestra.run.smithi044.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-08-24T09:50:40.974 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2023-08-24T09:50:40.974 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-08-24T09:50:40.975 INFO:teuthology.provision.fog.smithi044:Deploy complete!
2023-08-24T09:50:40.976 DEBUG:teuthology.parallel:result is None
2023-08-24T09:50:41.118 INFO:teuthology.lock.ops:Checking smithi044.front.sepia.ceph.com
2023-08-24T09:50:41.119 INFO:teuthology.lock.ops:New key found. Updating...
2023-08-24T09:50:41.148 INFO:teuthology.lock.ops:Updating [smithi044.front.sepia.ceph.com]: set os type and version on server
2023-08-24T09:50:41.149 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com'
2023-08-24T09:50:41.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-24T09:50:41.244 DEBUG:teuthology.orchestra.run.smithi044:> true
2023-08-24T09:50:41.312 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com'
2023-08-24T09:50:41.312 DEBUG:teuthology.orchestra.run.smithi044:> uname -m
2023-08-24T09:50:41.367 INFO:teuthology.orchestra.run.smithi044.stdout:x86_64
2023-08-24T09:50:41.368 DEBUG:teuthology.orchestra.run.smithi044:> cat /etc/os-release
2023-08-24T09:50:41.422 INFO:teuthology.orchestra.run.smithi044.stdout:NAME="CentOS Stream"
2023-08-24T09:50:41.422 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION="9"
2023-08-24T09:50:41.422 INFO:teuthology.orchestra.run.smithi044.stdout:ID="centos"
2023-08-24T09:50:41.423 INFO:teuthology.orchestra.run.smithi044.stdout:ID_LIKE="rhel fedora"
2023-08-24T09:50:41.423 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_ID="9"
2023-08-24T09:50:41.423 INFO:teuthology.orchestra.run.smithi044.stdout:PLATFORM_ID="platform:el9"
2023-08-24T09:50:41.423 INFO:teuthology.orchestra.run.smithi044.stdout:PRETTY_NAME="CentOS Stream 9"
2023-08-24T09:50:41.423 INFO:teuthology.orchestra.run.smithi044.stdout:ANSI_COLOR="0;31"
2023-08-24T09:50:41.423 INFO:teuthology.orchestra.run.smithi044.stdout:LOGO="fedora-logo-icon"
2023-08-24T09:50:41.423 INFO:teuthology.orchestra.run.smithi044.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2023-08-24T09:50:41.424 INFO:teuthology.orchestra.run.smithi044.stdout:HOME_URL="https://centos.org/"
2023-08-24T09:50:41.424 INFO:teuthology.orchestra.run.smithi044.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-08-24T09:50:41.424 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2023-08-24T09:50:41.424 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-08-24T09:50:41.425 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server
2023-08-24T09:50:41.447 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-24T09:50:41.541 INFO:teuthology.dispatcher.supervisor:Running job 7378495
2023-08-24T09:50:41.542 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/virtualenv/bin/teuthology -v --owner scheduled_matan@teuthology --archive /home/teuthworker/archive/matan-2023-08-24_09:25:19-crimson-rados-wip-matanb-crimson-testing-24.8-v2-distro-crimson-smithi/7378495 --name matan-2023-08-24_09:25:19-crimson-rados-wip-matanb-crimson-testing-24.8-v2-distro-crimson-smithi --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} -- /home/teuthworker/archive/matan-2023-08-24_09:25:19-crimson-rados-wip-matanb-crimson-testing-24.8-v2-distro-crimson-smithi/7378495/orig.config.yaml
2023-08-24T09:50:41.546 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/matan-2023-08-24_09:25:19-crimson-rados-wip-matanb-crimson-testing-24.8-v2-distro-crimson-smithi/7378495
2023-08-24T09:50:41.547 INFO:teuthology.dispatcher.supervisor:Job PID: 18752
2023-08-24T09:50:41.547 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-08-24T09:52:41.644 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-24T09:54:41.816 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-24T09:56:41.986 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-24T09:58:42.165 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-24T10:00:42.330 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-24T10:02:42.561 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-24T10:04:42.741 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-24T10:06:42.901 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-24T10:08:43.059 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-24T10:10:43.225 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-24T10:12:43.393 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-24T10:14:43.560 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-24T10:16:43.741 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-24T10:16:43.796 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120
2023-08-24T10:16:43.992 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-08-24T10:16:43.992 INFO:teuthology.nuke:Checking targets against current locks
2023-08-24T10:16:44.043 DEBUG:teuthology.nuke:shortname: smithi044
2023-08-24T10:16:44.044 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-24T10:16:44.067 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-08-24_09:25:19-crimson-rados-wip-matanb-crimson-testing-24.8-v2-distro-crimson-smithi/7378495', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-24 09:45:59.210161', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGU1O7ZRb845RA0Z29PFa3wn4bcxsCLcNP6wmrNVboK+s0EiUyaqMlCuaOEE1+Xdr6Xhv5IH04KyTRV2EA4nKQc='}
2023-08-24T10:16:44.118 INFO:teuthology.orchestra.console.smithi044:Power off
2023-08-24T10:16:44.119 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-24T10:16:44.140 DEBUG:teuthology.orchestra.console.smithi044:power off output: Chassis Power Control: Down/Off
2023-08-24T10:16:44.140 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-24T10:16:44.162 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is off
2023-08-24T10:16:44.263 INFO:teuthology.orchestra.console.smithi044:Power off completed
2023-08-24T10:16:44.412 INFO:teuthology.lock.ops:unlocked: smithi044.front.sepia.ceph.com