2023-11-10T20:31:36.832 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-10T20:31:36.834 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-10T20:31:36.947 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa69e9db670>, signals=[15]) 2023-11-10T20:31:36.948 INFO:teuthology.lock.ops:Start node 'smithi159.front.sepia.ceph.com' reimaging 2023-11-10T20:31:36.948 INFO:teuthology.lock.ops:Updating [smithi159.front.sepia.ceph.com]: reset os type and version on server 2023-11-10T20:31:36.949 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2023-11-10T20:31:36.968 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /host {"name": "smithi159"} 2023-11-10T20:31:37.044 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /image {"name": "smithi_ubuntu_20.04"} 2023-11-10T20:31:37.083 DEBUG:teuthology.provision.fog.smithi159:Requesting image smithi_ubuntu_20.04 (ID 2) 2023-11-10T20:31:37.084 DEBUG:teuthology.provision.fog.smithi159:FOG request: PUT /host/264 {"imageID": 2} 2023-11-10T20:31:37.136 INFO:teuthology.provision.fog.smithi159:Scheduling deploy of ubuntu 20.04 2023-11-10T20:31:37.137 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:31:37.231 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /tasktype {"name": "deploy"} 2023-11-10T20:31:37.284 DEBUG:teuthology.provision.fog.smithi159:FOG request: POST /host/264/task {"taskTypeID": 1} 2023-11-10T20:31:37.354 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:31:37.489 INFO:teuthology.orchestra.console.smithi159:Power off 2023-11-10T20:31:37.490 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-10T20:31:37.512 DEBUG:teuthology.orchestra.console.smithi159:power off output: Chassis Power Control: Down/Off 2023-11-10T20:31:37.513 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-10T20:31:37.536 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2023-11-10T20:31:41.540 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-10T20:31:41.650 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2023-11-10T20:31:45.654 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-10T20:31:45.767 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2023-11-10T20:31:49.771 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-10T20:31:49.880 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off 2023-11-10T20:31:49.981 INFO:teuthology.orchestra.console.smithi159:Power off completed 2023-11-10T20:31:50.082 INFO:teuthology.orchestra.console.smithi159:Power on 2023-11-10T20:31:50.082 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-10T20:31:50.104 DEBUG:teuthology.orchestra.console.smithi159:power on output: Chassis Power Control: Up/On 2023-11-10T20:31:50.105 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-10T20:31:50.127 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off 2023-11-10T20:31:54.130 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-10T20:31:54.239 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2023-11-10T20:31:54.340 INFO:teuthology.orchestra.console.smithi159:Power on completed 2023-11-10T20:31:54.442 INFO:teuthology.provision.fog.smithi159:Waiting for deploy to finish 2023-11-10T20:31:54.442 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:32:09.546 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:32:24.658 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:32:39.758 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:32:54.842 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:33:09.922 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:33:25.010 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:33:40.103 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:33:55.202 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:34:10.294 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:34:25.378 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:34:40.462 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:34:55.734 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:35:10.833 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:35:25.942 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:35:41.074 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None 2023-11-10T20:35:41.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-10T20:35:44.250 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159 2023-11-10T20:35:50.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-10T20:35:53.338 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159 2023-11-10T20:35:59.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-10T20:36:02.394 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159 2023-11-10T20:36:08.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-10T20:36:11.454 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159 2023-11-10T20:36:17.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-10T20:36:20.542 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159 2023-11-10T20:36:26.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-10T20:36:29.626 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159 2023-11-10T20:36:35.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-10T20:36:38.714 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159 2023-11-10T20:36:44.718 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-10T20:37:44.779 WARNING:teuthology.provision.fog.smithi159:timed out 2023-11-10T20:37:50.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-10T20:37:50.929 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-11-10T20:37:50.930 DEBUG:teuthology.orchestra.run.smithi159:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-11-10T20:37:51.450 INFO:teuthology.provision.fog.smithi159:Node is ready 2023-11-10T20:37:51.451 DEBUG:teuthology.orchestra.run.smithi159:> hostname 2023-11-10T20:37:51.457 INFO:teuthology.orchestra.run.smithi159.stdout:smithi159.front.sepia.ceph.com 2023-11-10T20:37:51.458 DEBUG:teuthology.orchestra.run.smithi159:> grep smithi159.front.sepia.ceph.com /etc/hosts 2023-11-10T20:37:51.464 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.15.159 smithi159.front.sepia.ceph.com smithi159 2023-11-10T20:37:51.464 DEBUG:teuthology.orchestra.run.smithi159:> sudo hostname smithi159 2023-11-10T20:37:51.588 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i -e 's/smithi159.front.sepia.ceph.com/smithi159/g' /etc/hosts 2023-11-10T20:37:51.604 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i -e 's/172.21.15.159/172.21.15.159/g' /etc/hosts 2023-11-10T20:37:51.615 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i -e 's/smithi159.front.sepia.ceph.com/smithi159/g' /etc/hostname 2023-11-10T20:37:51.628 DEBUG:teuthology.orchestra.run.smithi159:> sudo hostname smithi159 2023-11-10T20:37:51.690 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release 2023-11-10T20:37:51.694 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="Ubuntu" 2023-11-10T20:37:51.694 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-11-10T20:37:51.695 INFO:teuthology.orchestra.run.smithi159.stdout:ID=ubuntu 2023-11-10T20:37:51.695 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE=debian 2023-11-10T20:37:51.695 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-11-10T20:37:51.695 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="20.04" 2023-11-10T20:37:51.695 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-10T20:37:51.696 INFO:teuthology.orchestra.run.smithi159.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-10T20:37:51.696 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-10T20:37:51.696 INFO:teuthology.orchestra.run.smithi159.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-10T20:37:51.696 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_CODENAME=focal 2023-11-10T20:37:51.696 INFO:teuthology.orchestra.run.smithi159.stdout:UBUNTU_CODENAME=focal 2023-11-10T20:37:51.697 INFO:teuthology.provision.fog.smithi159:Deploy complete! 2023-11-10T20:37:51.698 DEBUG:teuthology.parallel:result is None 2023-11-10T20:37:51.876 INFO:teuthology.lock.ops:Checking smithi159.front.sepia.ceph.com 2023-11-10T20:37:51.877 INFO:teuthology.lock.ops:New key found. Updating... 2023-11-10T20:37:51.903 INFO:teuthology.lock.ops:Updating [smithi159.front.sepia.ceph.com]: set os type and version on server 2023-11-10T20:37:51.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2023-11-10T20:37:51.905 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-10T20:37:51.954 DEBUG:teuthology.orchestra.run.smithi159:> true 2023-11-10T20:37:52.076 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2023-11-10T20:37:52.077 DEBUG:teuthology.orchestra.run.smithi159:> uname -m 2023-11-10T20:37:52.083 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2023-11-10T20:37:52.084 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release 2023-11-10T20:37:52.090 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="Ubuntu" 2023-11-10T20:37:52.090 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-11-10T20:37:52.091 INFO:teuthology.orchestra.run.smithi159.stdout:ID=ubuntu 2023-11-10T20:37:52.091 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE=debian 2023-11-10T20:37:52.091 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-11-10T20:37:52.092 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="20.04" 2023-11-10T20:37:52.092 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-10T20:37:52.092 INFO:teuthology.orchestra.run.smithi159.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-10T20:37:52.092 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-10T20:37:52.093 INFO:teuthology.orchestra.run.smithi159.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-10T20:37:52.093 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_CODENAME=focal 2023-11-10T20:37:52.093 INFO:teuthology.orchestra.run.smithi159.stdout:UBUNTU_CODENAME=focal 2023-11-10T20:37:52.094 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2023-11-10T20:37:52.119 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-10T20:37:52.235 INFO:teuthology.dispatcher.supervisor:Running job 7454548 2023-11-10T20:37:52.235 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-11-10_18:18:41-rados-wip-yuri3-testing-2023-11-09-1355-quincy-distro-default-smithi/7454548 --name yuriw-2023-11-10_18:18:41-rados-wip-yuri3-testing-2023-11-09-1355-quincy-distro-default-smithi --description rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} -- /home/teuthworker/archive/yuriw-2023-11-10_18:18:41-rados-wip-yuri3-testing-2023-11-09-1355-quincy-distro-default-smithi/7454548/orig.config.yaml 2023-11-10T20:37:52.238 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-11-10_18:18:41-rados-wip-yuri3-testing-2023-11-09-1355-quincy-distro-default-smithi/7454548 2023-11-10T20:37:52.239 INFO:teuthology.dispatcher.supervisor:Job PID: 3678226 2023-11-10T20:37:52.239 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-11-10T20:39:52.339 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-10T20:41:52.511 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-10T20:43:52.671 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-10T20:45:52.839 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-10T20:47:53.007 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-10T20:49:53.173 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-10T20:51:53.359 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-10T20:53:53.531 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-10T20:53:53.601 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120 2023-11-10T20:53:53.902 INFO:teuthology.dispatcher.supervisor:Unlocking machines... 2023-11-10T20:53:53.949 INFO:teuthology.lock.ops:unlocked: smithi159.front.sepia.ceph.com