2023-07-23T07:27:26.386 INFO:root:teuthology version: 0.0.1.dev158+g1426592
2023-07-23T07:27:26.387 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-23T07:27:26.388 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-23T07:27:26.514 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f1372f16e20>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f13744181f0>, signals=[15])
2023-07-23T07:27:26.516 INFO:teuthology.lock.ops:Start node 'smithi089.front.sepia.ceph.com' reimaging
2023-07-23T07:27:26.517 INFO:teuthology.lock.ops:Updating [smithi089.front.sepia.ceph.com]: reset os type and version on server
2023-07-23T07:27:26.517 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server
2023-07-23T07:27:26.545 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /host {"name": "smithi089"}
2023-07-23T07:27:26.725 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-07-23T07:27:26.768 DEBUG:teuthology.provision.fog.smithi089:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-07-23T07:27:26.768 DEBUG:teuthology.provision.fog.smithi089:FOG request: PUT /host/194 {"imageID": 2}
2023-07-23T07:27:26.815 INFO:teuthology.provision.fog.smithi089:Scheduling deploy of ubuntu 20.04
2023-07-23T07:27:26.815 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:27:26.856 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /tasktype {"name": "deploy"}
2023-07-23T07:27:26.892 DEBUG:teuthology.provision.fog.smithi089:FOG request: POST /host/194/task {"taskTypeID": 1}
2023-07-23T07:27:27.000 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:27:27.087 INFO:teuthology.orchestra.console:Power off smithi089
2023-07-23T07:27:27.088 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-23T07:27:27.337 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-23T07:27:27.338 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-23T07:27:27.362 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-23T07:27:31.366 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-23T07:27:31.476 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-23T07:27:35.481 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-23T07:27:35.591 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-23T07:27:35.691 INFO:teuthology.orchestra.console:Power off for smithi089 completed
2023-07-23T07:27:35.792 INFO:teuthology.orchestra.console:Power on smithi089
2023-07-23T07:27:35.792 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-23T07:27:35.811 DEBUG:teuthology.orchestra.console:power on output: Chassis Power Control: Up/On
2023-07-23T07:27:35.812 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-23T07:27:35.829 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-23T07:27:39.834 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-23T07:27:39.946 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-23T07:27:43.950 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-23T07:27:44.061 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-23T07:27:44.162 INFO:teuthology.orchestra.console:Power on for smithi089 completed
2023-07-23T07:27:44.263 INFO:teuthology.provision.fog.smithi089:Waiting for deploy to finish
2023-07-23T07:27:44.263 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:27:59.328 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:28:14.398 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:28:29.466 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:28:44.533 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:28:59.602 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:29:14.672 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:29:29.737 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:29:44.795 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:29:59.861 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:30:14.912 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:30:29.975 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:30:45.043 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:31:00.104 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:31:15.158 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:31:30.210 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None
2023-07-23T07:31:30.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-23T07:31:48.720 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89
2023-07-23T07:31:54.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-23T07:31:57.808 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89
2023-07-23T07:32:03.813 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-23T07:32:06.864 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89
2023-07-23T07:32:12.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-23T07:32:15.920 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89
2023-07-23T07:32:21.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-23T07:32:25.008 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89
2023-07-23T07:32:31.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-23T07:33:03.120 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-23T07:33:03.121 DEBUG:teuthology.orchestra.run.smithi089:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-23T07:33:24.219 INFO:teuthology.provision.fog.smithi089:Node is ready
2023-07-23T07:33:24.220 DEBUG:teuthology.orchestra.run.smithi089:> hostname
2023-07-23T07:33:24.227 INFO:teuthology.orchestra.run.smithi089.stdout:smithi089.front.sepia.ceph.com
2023-07-23T07:33:24.227 DEBUG:teuthology.orchestra.run.smithi089:> grep smithi089.front.sepia.ceph.com /etc/hosts
2023-07-23T07:33:24.274 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89 smithi089.front.sepia.ceph.com smithi089
2023-07-23T07:33:24.275 DEBUG:teuthology.orchestra.run.smithi089:> sudo hostname smithi089
2023-07-23T07:33:24.329 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i -e 's/smithi089.front.sepia.ceph.com/smithi089/g' /etc/hosts
2023-07-23T07:33:24.344 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i -e 's/172.21.15.89/172.21.15.89/g' /etc/hosts
2023-07-23T07:33:24.357 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i -e 's/smithi089.front.sepia.ceph.com/smithi089/g' /etc/hostname
2023-07-23T07:33:24.370 DEBUG:teuthology.orchestra.run.smithi089:> sudo hostname smithi089
2023-07-23T07:33:24.383 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release
2023-07-23T07:33:24.429 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="Ubuntu"
2023-07-23T07:33:24.430 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="20.04.4 LTS (Focal Fossa)"
2023-07-23T07:33:24.430 INFO:teuthology.orchestra.run.smithi089.stdout:ID=ubuntu
2023-07-23T07:33:24.430 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE=debian
2023-07-23T07:33:24.430 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS"
2023-07-23T07:33:24.430 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="20.04"
2023-07-23T07:33:24.430 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://www.ubuntu.com/"
2023-07-23T07:33:24.431 INFO:teuthology.orchestra.run.smithi089.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-07-23T07:33:24.431 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-07-23T07:33:24.431 INFO:teuthology.orchestra.run.smithi089.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-07-23T07:33:24.431 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_CODENAME=focal
2023-07-23T07:33:24.431 INFO:teuthology.orchestra.run.smithi089.stdout:UBUNTU_CODENAME=focal
2023-07-23T07:33:24.432 INFO:teuthology.provision.fog.smithi089:Deploy complete!
2023-07-23T07:33:24.433 DEBUG:teuthology.parallel:result is None
2023-07-23T07:33:24.587 INFO:teuthology.lock.ops:Checking smithi089.front.sepia.ceph.com
2023-07-23T07:33:24.590 INFO:teuthology.lock.ops:Updating [smithi089.front.sepia.ceph.com]: set os type and version on server
2023-07-23T07:33:24.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com'
2023-07-23T07:33:24.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-23T07:33:24.636 DEBUG:teuthology.orchestra.run.smithi089:> true
2023-07-23T07:33:24.761 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com'
2023-07-23T07:33:24.761 DEBUG:teuthology.orchestra.run.smithi089:> uname -m
2023-07-23T07:33:24.767 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64
2023-07-23T07:33:24.768 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release
2023-07-23T07:33:24.773 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="Ubuntu"
2023-07-23T07:33:24.774 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="20.04.4 LTS (Focal Fossa)"
2023-07-23T07:33:24.774 INFO:teuthology.orchestra.run.smithi089.stdout:ID=ubuntu
2023-07-23T07:33:24.774 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE=debian
2023-07-23T07:33:24.774 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS"
2023-07-23T07:33:24.774 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="20.04"
2023-07-23T07:33:24.774 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://www.ubuntu.com/"
2023-07-23T07:33:24.774 INFO:teuthology.orchestra.run.smithi089.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-07-23T07:33:24.775 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-07-23T07:33:24.775 INFO:teuthology.orchestra.run.smithi089.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-07-23T07:33:24.775 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_CODENAME=focal
2023-07-23T07:33:24.775 INFO:teuthology.orchestra.run.smithi089.stdout:UBUNTU_CODENAME=focal
2023-07-23T07:33:24.775 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server
2023-07-23T07:33:24.808 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-23T07:33:24.809 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-23T07:33:24.898 INFO:teuthology.dispatcher.supervisor:Running job 7348165
2023-07-23T07:33:24.898 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_1426592083662fef3e03bbeec33624331f0fdbc1/virtualenv/bin/teuthology -v --owner scheduled_jcollin@teuthology --archive /home/teuthworker/archive/jcollin-2023-07-23_07:26:16-fs:mirror-ha-wip-cephfs-mirror-fix_quincy-distro-default-smithi/7348165 --name jcollin-2023-07-23_07:26:16-fs:mirror-ha-wip-cephfs-mirror-fix_quincy-distro-default-smithi --description fs:mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/0-cephfs-mirror-ha-workunit} -- /home/teuthworker/archive/jcollin-2023-07-23_07:26:16-fs:mirror-ha-wip-cephfs-mirror-fix_quincy-distro-default-smithi/7348165/orig.config.yaml
2023-07-23T07:33:24.902 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/jcollin-2023-07-23_07:26:16-fs:mirror-ha-wip-cephfs-mirror-fix_quincy-distro-default-smithi/7348165
2023-07-23T07:33:24.903 INFO:teuthology.dispatcher.supervisor:Job PID: 26938
2023-07-23T07:33:24.903 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-07-23T07:35:24.909 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-23T07:35:24.912 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-23T07:37:25.151 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-23T07:37:25.151 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-23T07:39:25.282 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-23T07:39:25.283 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-23T07:41:25.434 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-23T07:41:25.436 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-23T07:43:25.544 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-23T07:43:25.545 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-23T07:45:25.680 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-23T07:45:25.681 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-23T07:47:25.841 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-23T07:47:25.842 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-23T07:49:25.962 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-23T07:49:25.963 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-23T07:51:26.031 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-23T07:51:26.032 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-23T07:51:26.101 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120
2023-07-23T07:51:26.187 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-07-23T07:51:26.187 INFO:teuthology.nuke:Checking targets against current locks
2023-07-23T07:51:26.238 DEBUG:teuthology.nuke:shortname: smithi089
2023-07-23T07:51:26.238 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-23T07:51:26.264 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/jcollin-2023-07-23_07:26:16-fs:mirror-ha-wip-cephfs-mirror-fix_quincy-distro-default-smithi/7348165', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-23 07:27:25.656221', 'locked_by': 'scheduled_jcollin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='}
2023-07-23T07:51:26.319 INFO:teuthology.orchestra.console:Power off smithi089
2023-07-23T07:51:26.320 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-23T07:51:26.339 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-23T07:51:26.340 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-23T07:51:26.358 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-23T07:51:26.459 INFO:teuthology.orchestra.console:Power off for smithi089 completed
2023-07-23T07:51:26.617 INFO:teuthology.lock.ops:unlocked: smithi089.front.sepia.ceph.com