2023-08-10T06:59:55.529 INFO:root:teuthology version: 0.0.1.dev181+g7fda959 2023-08-10T06:59:55.531 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T06:59:55.636 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f65319cedc0>, signals=[15]) 2023-08-10T06:59:55.638 INFO:teuthology.lock.ops:Start node 'smithi040.front.sepia.ceph.com' reimaging 2023-08-10T06:59:55.638 INFO:teuthology.lock.ops:Updating [smithi040.front.sepia.ceph.com]: reset os type and version on server 2023-08-10T06:59:55.638 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2023-08-10T06:59:55.654 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /host {"name": "smithi040"} 2023-08-10T06:59:55.723 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-08-10T06:59:55.762 DEBUG:teuthology.provision.fog.smithi040:Requesting image smithi_centos_8.stream (ID 3) 2023-08-10T06:59:55.762 DEBUG:teuthology.provision.fog.smithi040:FOG request: PUT /host/145 {"imageID": 3} 2023-08-10T06:59:55.819 INFO:teuthology.provision.fog.smithi040:Scheduling deploy of centos 8.stream 2023-08-10T06:59:55.819 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T06:59:55.913 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /tasktype {"name": "deploy"} 2023-08-10T06:59:55.944 DEBUG:teuthology.provision.fog.smithi040:FOG request: POST /host/145/task {"taskTypeID": 1} 2023-08-10T06:59:55.995 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T06:59:56.127 INFO:teuthology.orchestra.console.smithi040:Power off 2023-08-10T06:59:56.128 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-10T06:59:56.150 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2023-08-10T06:59:56.151 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-10T06:59:56.171 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2023-08-10T07:00:00.173 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-10T07:00:00.283 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2023-08-10T07:00:04.288 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-10T07:00:04.400 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2023-08-10T07:00:08.404 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-10T07:00:08.515 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2023-08-10T07:00:08.616 INFO:teuthology.orchestra.console.smithi040:Power off completed 2023-08-10T07:00:08.717 INFO:teuthology.orchestra.console.smithi040:Power on 2023-08-10T07:00:08.718 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-08-10T07:00:08.739 DEBUG:teuthology.orchestra.console.smithi040:power on output: Chassis Power Control: Up/On 2023-08-10T07:00:08.739 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-10T07:00:08.761 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2023-08-10T07:00:12.764 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-10T07:00:12.873 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2023-08-10T07:00:12.974 INFO:teuthology.orchestra.console.smithi040:Power on completed 2023-08-10T07:00:13.075 INFO:teuthology.provision.fog.smithi040:Waiting for deploy to finish 2023-08-10T07:00:13.075 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:00:28.285 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:00:43.513 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:00:58.708 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:01:13.912 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:01:29.151 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:01:44.425 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:01:59.715 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:02:14.932 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:02:30.188 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:02:45.504 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:03:00.772 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:03:16.180 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:03:31.395 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:03:46.608 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:04:01.847 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:04:17.076 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:04:32.300 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:04:47.504 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:05:02.718 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:05:17.904 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:05:33.222 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:05:48.461 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:06:03.680 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:06:18.873 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:06:34.064 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:06:49.267 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:07:04.440 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:07:19.589 DEBUG:teuthology.provision.fog.smithi040:FOG request: GET /task/active None 2023-08-10T07:07:19.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-10T07:07:22.800 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-08-10T07:07:28.804 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-10T07:07:31.856 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-08-10T07:07:37.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-10T07:07:40.944 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-08-10T07:07:46.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-10T07:07:50.032 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-08-10T07:07:56.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-10T07:07:59.120 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-08-10T07:08:05.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-10T07:08:08.208 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-08-10T07:08:14.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-10T07:09:14.228 WARNING:teuthology.provision.fog.smithi040:timed out 2023-08-10T07:09:20.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-10T07:09:20.481 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-08-10T07:09:20.482 DEBUG:teuthology.orchestra.run.smithi040:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-08-10T07:09:20.734 INFO:teuthology.provision.fog.smithi040:Node is ready 2023-08-10T07:09:20.735 DEBUG:teuthology.orchestra.run.smithi040:> hostname 2023-08-10T07:09:20.755 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040.front.sepia.ceph.com 2023-08-10T07:09:20.755 DEBUG:teuthology.orchestra.run.smithi040:> grep smithi040.front.sepia.ceph.com /etc/hosts 2023-08-10T07:09:20.814 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40 smithi040.front.sepia.ceph.com smithi040 2023-08-10T07:09:20.815 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname smithi040 2023-08-10T07:09:20.917 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i -e 's/smithi040.front.sepia.ceph.com/smithi040/g' /etc/hosts 2023-08-10T07:09:20.948 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i -e 's/172.21.15.40/172.21.15.40/g' /etc/hosts 2023-08-10T07:09:21.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i -e 's/smithi040.front.sepia.ceph.com/smithi040/g' /etc/hostname 2023-08-10T07:09:21.086 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname smithi040 2023-08-10T07:09:21.154 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2023-08-10T07:09:21.212 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2023-08-10T07:09:21.212 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="8" 2023-08-10T07:09:21.213 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2023-08-10T07:09:21.213 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2023-08-10T07:09:21.213 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="8" 2023-08-10T07:09:21.213 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el8" 2023-08-10T07:09:21.213 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 8" 2023-08-10T07:09:21.213 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2023-08-10T07:09:21.213 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-08-10T07:09:21.214 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2023-08-10T07:09:21.214 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-10T07:09:21.214 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-10T07:09:21.214 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-08-10T07:09:21.215 INFO:teuthology.provision.fog.smithi040:Deploy complete! 2023-08-10T07:09:21.216 DEBUG:teuthology.parallel:result is None 2023-08-10T07:09:21.321 INFO:teuthology.lock.ops:Checking smithi040.front.sepia.ceph.com 2023-08-10T07:09:21.322 INFO:teuthology.lock.ops:New key found. Updating... 2023-08-10T07:09:21.351 INFO:teuthology.lock.ops:Updating [smithi040.front.sepia.ceph.com]: set os type and version on server 2023-08-10T07:09:21.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-08-10T07:09:21.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-10T07:09:21.418 DEBUG:teuthology.orchestra.run.smithi040:> true 2023-08-10T07:09:21.490 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-08-10T07:09:21.491 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2023-08-10T07:09:21.548 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2023-08-10T07:09:21.549 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2023-08-10T07:09:21.606 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2023-08-10T07:09:21.606 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="8" 2023-08-10T07:09:21.606 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2023-08-10T07:09:21.606 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2023-08-10T07:09:21.607 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="8" 2023-08-10T07:09:21.607 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el8" 2023-08-10T07:09:21.607 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 8" 2023-08-10T07:09:21.607 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2023-08-10T07:09:21.607 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-08-10T07:09:21.607 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2023-08-10T07:09:21.607 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-10T07:09:21.607 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-10T07:09:21.608 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-08-10T07:09:21.608 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2023-08-10T07:09:21.633 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:09:21.715 INFO:teuthology.dispatcher.supervisor:Running job 7364470 2023-08-10T07:09:21.715 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/bin/teuthology -v --owner scheduled_yingxin@teuthology --archive /home/teuthworker/archive/yingxin-2023-08-10_06:58:49-crimson-rados-wip-yingxin-crimson-multi-core-msgr-UT-distro-default-smithi/7364470 --name yingxin-2023-08-10_06:58:49-crimson-rados-wip-yingxin-crimson-multi-core-msgr-UT-distro-default-smithi --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_8 crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} -- /home/teuthworker/archive/yingxin-2023-08-10_06:58:49-crimson-rados-wip-yingxin-crimson-multi-core-msgr-UT-distro-default-smithi/7364470/orig.config.yaml 2023-08-10T07:09:21.719 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yingxin-2023-08-10_06:58:49-crimson-rados-wip-yingxin-crimson-multi-core-msgr-UT-distro-default-smithi/7364470 2023-08-10T07:09:21.720 INFO:teuthology.dispatcher.supervisor:Job PID: 23201 2023-08-10T07:09:21.720 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-08-10T07:11:21.821 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:13:22.008 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:15:22.177 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:17:22.373 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:19:22.534 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:21:22.712 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:23:22.886 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:25:23.057 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:27:23.246 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:29:23.421 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:31:23.601 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:33:23.769 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:35:23.937 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:37:24.117 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:39:24.272 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:41:24.449 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:43:24.613 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:45:24.769 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:47:24.925 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:49:25.086 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-10T07:49:25.138 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120 2023-08-10T07:49:25.214 INFO:teuthology.dispatcher.supervisor:Unlocking machines... 2023-08-10T07:49:25.254 INFO:teuthology.lock.ops:unlocked: smithi040.front.sepia.ceph.com