2023-07-24T07:16:19.660 INFO:root:teuthology version: 0.0.1.dev158+g1426592
2023-07-24T07:16:19.662 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T07:16:19.662 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T07:16:19.776 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f48641aecd0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f48656ae1f0>, signals=[15])
2023-07-24T07:16:19.777 INFO:teuthology.lock.ops:Start node 'smithi122.front.sepia.ceph.com' reimaging
2023-07-24T07:16:19.778 INFO:teuthology.lock.ops:Updating [smithi122.front.sepia.ceph.com]: reset os type and version on server
2023-07-24T07:16:19.778 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server
2023-07-24T07:16:19.801 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /host {"name": "smithi122"}
2023-07-24T07:16:19.869 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-07-24T07:16:19.901 DEBUG:teuthology.provision.fog.smithi122:Requesting image smithi_centos_8.stream (ID 3)
2023-07-24T07:16:19.902 DEBUG:teuthology.provision.fog.smithi122:FOG request: PUT /host/227 {"imageID": 3}
2023-07-24T07:16:19.945 INFO:teuthology.provision.fog.smithi122:Scheduling deploy of centos 8.stream
2023-07-24T07:16:19.945 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T07:16:19.991 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /tasktype {"name": "deploy"}
2023-07-24T07:16:20.026 DEBUG:teuthology.provision.fog.smithi122:FOG request: POST /host/227/task {"taskTypeID": 1}
2023-07-24T07:16:20.090 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T07:16:20.182 INFO:teuthology.orchestra.console:Power off smithi122
2023-07-24T07:16:20.183 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-24T07:16:20.204 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-24T07:16:20.205 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T07:16:20.225 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-24T07:16:20.326 INFO:teuthology.orchestra.console:Power off for smithi122 completed
2023-07-24T07:16:20.426 INFO:teuthology.orchestra.console:Power on smithi122
2023-07-24T07:16:20.427 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-24T07:16:20.447 DEBUG:teuthology.orchestra.console:power on output: Chassis Power Control: Up/On
2023-07-24T07:16:20.448 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T07:16:20.468 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-24T07:16:24.473 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T07:16:24.582 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-24T07:16:28.587 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T07:16:28.696 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-24T07:16:28.797 INFO:teuthology.orchestra.console:Power on for smithi122 completed
2023-07-24T07:16:28.898 INFO:teuthology.provision.fog.smithi122:Waiting for deploy to finish
2023-07-24T07:16:28.898 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T07:16:44.015 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T07:16:59.138 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T07:17:14.286 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T07:17:29.427 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T07:17:44.576 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T07:17:59.756 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T07:18:14.988 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T07:18:30.333 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T07:18:45.584 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T07:19:00.953 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T07:19:16.145 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T07:19:16.418 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T07:19:34.768 WARNING:teuthology.provision.fog.smithi122:[Errno None] Unable to connect to port 22 on 172.21.15.122
2023-07-24T07:19:40.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T07:19:43.824 WARNING:teuthology.provision.fog.smithi122:[Errno None] Unable to connect to port 22 on 172.21.15.122
2023-07-24T07:19:49.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T07:19:52.912 WARNING:teuthology.provision.fog.smithi122:[Errno None] Unable to connect to port 22 on 172.21.15.122
2023-07-24T07:19:58.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T07:20:02.000 WARNING:teuthology.provision.fog.smithi122:[Errno None] Unable to connect to port 22 on 172.21.15.122
2023-07-24T07:20:08.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T07:21:08.068 WARNING:teuthology.provision.fog.smithi122:timed out
2023-07-24T07:21:14.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T07:21:14.282 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-24T07:21:14.283 DEBUG:teuthology.orchestra.run.smithi122:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-24T07:21:14.504 INFO:teuthology.provision.fog.smithi122:Node is ready
2023-07-24T07:21:14.504 DEBUG:teuthology.orchestra.run.smithi122:> hostname
2023-07-24T07:21:14.566 INFO:teuthology.orchestra.run.smithi122.stdout:smithi122.front.sepia.ceph.com
2023-07-24T07:21:14.566 DEBUG:teuthology.orchestra.run.smithi122:> grep smithi122.front.sepia.ceph.com /etc/hosts
2023-07-24T07:21:14.627 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.15.122 smithi122.front.sepia.ceph.com smithi122
2023-07-24T07:21:14.628 DEBUG:teuthology.orchestra.run.smithi122:> sudo hostname smithi122
2023-07-24T07:21:14.745 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i -e 's/smithi122.front.sepia.ceph.com/smithi122/g' /etc/hosts
2023-07-24T07:21:14.780 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i -e 's/172.21.15.122/172.21.15.122/g' /etc/hosts
2023-07-24T07:21:14.854 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i -e 's/smithi122.front.sepia.ceph.com/smithi122/g' /etc/hostname
2023-07-24T07:21:14.928 DEBUG:teuthology.orchestra.run.smithi122:> sudo hostname smithi122
2023-07-24T07:21:15.001 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release
2023-07-24T07:21:15.063 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream"
2023-07-24T07:21:15.063 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="8"
2023-07-24T07:21:15.063 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos"
2023-07-24T07:21:15.063 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora"
2023-07-24T07:21:15.063 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="8"
2023-07-24T07:21:15.064 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el8"
2023-07-24T07:21:15.064 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 8"
2023-07-24T07:21:15.064 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31"
2023-07-24T07:21:15.064 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-07-24T07:21:15.064 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/"
2023-07-24T07:21:15.064 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-24T07:21:15.064 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-24T07:21:15.065 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-07-24T07:21:15.065 INFO:teuthology.provision.fog.smithi122:Deploy complete!
2023-07-24T07:21:15.066 DEBUG:teuthology.parallel:result is None
2023-07-24T07:21:15.185 INFO:teuthology.lock.ops:Checking smithi122.front.sepia.ceph.com
2023-07-24T07:21:15.186 INFO:teuthology.lock.ops:New key found. Updating...
2023-07-24T07:21:15.207 INFO:teuthology.lock.ops:Updating [smithi122.front.sepia.ceph.com]: set os type and version on server
2023-07-24T07:21:15.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com'
2023-07-24T07:21:15.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T07:21:15.273 DEBUG:teuthology.orchestra.run.smithi122:> true
2023-07-24T07:21:15.346 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com'
2023-07-24T07:21:15.346 DEBUG:teuthology.orchestra.run.smithi122:> uname -m
2023-07-24T07:21:15.406 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64
2023-07-24T07:21:15.406 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release
2023-07-24T07:21:15.468 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream"
2023-07-24T07:21:15.468 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="8"
2023-07-24T07:21:15.468 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos"
2023-07-24T07:21:15.469 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora"
2023-07-24T07:21:15.469 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="8"
2023-07-24T07:21:15.469 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el8"
2023-07-24T07:21:15.469 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 8"
2023-07-24T07:21:15.469 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31"
2023-07-24T07:21:15.469 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-07-24T07:21:15.469 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/"
2023-07-24T07:21:15.470 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-24T07:21:15.470 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-24T07:21:15.470 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-07-24T07:21:15.470 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server
2023-07-24T07:21:15.497 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T07:21:15.498 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T07:21:15.597 INFO:teuthology.dispatcher.supervisor:Running job 7349286
2023-07-24T07:21:15.598 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_1426592083662fef3e03bbeec33624331f0fdbc1/virtualenv/bin/teuthology -v --owner scheduled_matan@teuthology --archive /home/teuthworker/archive/matan-2023-07-24_07:15:05-crimson-rados-wip-matanb-crimson-only-61943-distro-crimson-smithi/7349286 --name matan-2023-07-24_07:15:05-crimson-rados-wip-matanb-crimson-only-61943-distro-crimson-smithi --description crimson-rados/perf/{centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} -- /home/teuthworker/archive/matan-2023-07-24_07:15:05-crimson-rados-wip-matanb-crimson-only-61943-distro-crimson-smithi/7349286/orig.config.yaml
2023-07-24T07:21:15.601 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/matan-2023-07-24_07:15:05-crimson-rados-wip-matanb-crimson-only-61943-distro-crimson-smithi/7349286
2023-07-24T07:21:15.601 INFO:teuthology.dispatcher.supervisor:Job PID: 31576
2023-07-24T07:21:15.602 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-07-24T07:23:15.693 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T07:23:15.694 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T07:25:15.876 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T07:25:15.877 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T07:27:16.049 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T07:27:16.050 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T07:29:16.226 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T07:29:16.228 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T07:31:16.401 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T07:31:16.401 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T07:33:16.589 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T07:33:16.591 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T07:35:16.920 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T07:35:16.922 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T07:37:17.140 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T07:37:17.141 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T07:39:17.322 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T07:39:17.323 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T07:41:17.524 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T07:41:17.525 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T07:43:17.705 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T07:43:17.705 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T07:43:17.775 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120
2023-07-24T07:43:17.855 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-07-24T07:43:17.856 INFO:teuthology.nuke:Checking targets against current locks
2023-07-24T07:43:17.906 DEBUG:teuthology.nuke:shortname: smithi122
2023-07-24T07:43:17.906 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-24T07:43:17.930 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-07-24_07:15:05-crimson-rados-wip-matanb-crimson-only-61943-distro-crimson-smithi/7349286', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-24 07:16:18.942306', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPE/1+RvHOj2Pyf5hVeg/nEi1NwmXY6dE822FceUWcI+WnI78CdSn6CfwzsYlbOy8O+w2ggKuFR1pLVotCPJakw='}
2023-07-24T07:43:18.000 INFO:teuthology.orchestra.console:Power off smithi122
2023-07-24T07:43:18.001 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-24T07:43:18.022 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-24T07:43:18.023 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T07:43:18.045 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-24T07:43:18.146 INFO:teuthology.orchestra.console:Power off for smithi122 completed
2023-07-24T07:43:18.325 INFO:teuthology.lock.ops:unlocked: smithi122.front.sepia.ceph.com