2023-07-05T14:56:54.023 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-05T14:56:54.026 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T14:56:54.027 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T14:56:54.245 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcd9ce6c670>, signals=[15]) 2023-07-05T14:56:54.246 INFO:teuthology.lock.ops:Start node 'smithi089.front.sepia.ceph.com' reimaging 2023-07-05T14:56:54.247 INFO:teuthology.lock.ops:Updating [smithi089.front.sepia.ceph.com]: reset os type and version on server 2023-07-05T14:56:54.247 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2023-07-05T14:56:54.286 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /host {"name": "smithi089"} 2023-07-05T14:56:54.397 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /image {"name": "smithi_ubuntu_22.04"} 2023-07-05T14:56:54.467 DEBUG:teuthology.provision.fog.smithi089:Requesting image smithi_ubuntu_22.04 (ID 6) 2023-07-05T14:56:54.467 DEBUG:teuthology.provision.fog.smithi089:FOG request: PUT /host/194 {"imageID": 6} 2023-07-05T14:56:54.533 INFO:teuthology.provision.fog.smithi089:Scheduling deploy of ubuntu 22.04 2023-07-05T14:56:54.533 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T14:56:54.630 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /tasktype {"name": "deploy"} 2023-07-05T14:56:54.677 DEBUG:teuthology.provision.fog.smithi089:FOG request: POST /host/194/task {"taskTypeID": 1} 2023-07-05T14:56:54.761 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T14:56:54.927 INFO:teuthology.orchestra.console:Power off smithi089 2023-07-05T14:56:54.928 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-05T14:56:55.005 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-05T14:56:59.047 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-05T14:57:03.170 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-05T14:57:03.386 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2023-07-05T14:57:03.491 INFO:teuthology.orchestra.console:Power on smithi089 2023-07-05T14:57:03.492 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-07-05T14:57:03.527 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-05T14:57:07.560 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-05T14:57:07.785 INFO:teuthology.orchestra.console:Power on for smithi089 completed 2023-07-05T14:57:07.890 INFO:teuthology.provision.fog.smithi089:Waiting for deploy to finish 2023-07-05T14:57:07.891 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T14:57:22.969 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T14:57:38.053 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T14:57:53.125 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T14:58:08.196 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T14:58:23.266 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T14:58:38.336 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T14:58:53.404 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T14:59:08.468 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T14:59:23.546 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T14:59:38.620 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T14:59:53.692 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T15:00:08.760 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T15:00:23.832 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-05T15:00:23.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-05T15:00:26.960 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-07-05T15:00:32.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-05T15:00:36.048 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-07-05T15:00:42.052 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-05T15:00:45.104 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-07-05T15:00:51.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-05T15:00:54.164 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-07-05T15:01:00.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-05T15:01:03.248 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-07-05T15:01:09.253 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-05T15:01:12.304 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-07-05T15:01:18.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-05T15:02:18.369 WARNING:teuthology.provision.fog.smithi089:timed out 2023-07-05T15:02:24.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-05T15:02:24.647 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-07-05T15:02:24.648 DEBUG:teuthology.orchestra.run.smithi089:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-07-05T15:02:25.124 INFO:teuthology.provision.fog.smithi089:Node is ready 2023-07-05T15:02:25.125 DEBUG:teuthology.orchestra.run.smithi089:> hostname 2023-07-05T15:02:25.173 INFO:teuthology.orchestra.run.smithi089.stdout:smithi089.front.sepia.ceph.com 2023-07-05T15:02:25.174 DEBUG:teuthology.orchestra.run.smithi089:> grep smithi089.front.sepia.ceph.com /etc/hosts 2023-07-05T15:02:25.221 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89 smithi089.front.sepia.ceph.com smithi089 2023-07-05T15:02:25.222 DEBUG:teuthology.orchestra.run.smithi089:> sudo hostname smithi089 2023-07-05T15:02:25.362 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i -e 's/smithi089.front.sepia.ceph.com/smithi089/g' /etc/hosts 2023-07-05T15:02:25.377 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i -e 's/172.21.15.89/172.21.15.89/g' /etc/hosts 2023-07-05T15:02:25.438 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i -e 's/smithi089.front.sepia.ceph.com/smithi089/g' /etc/hostname 2023-07-05T15:02:25.493 DEBUG:teuthology.orchestra.run.smithi089:> sudo hostname smithi089 2023-07-05T15:02:25.553 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2023-07-05T15:02:25.605 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-07-05T15:02:25.606 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="Ubuntu" 2023-07-05T15:02:25.606 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="22.04" 2023-07-05T15:02:25.606 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-07-05T15:02:25.606 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_CODENAME=jammy 2023-07-05T15:02:25.606 INFO:teuthology.orchestra.run.smithi089.stdout:ID=ubuntu 2023-07-05T15:02:25.606 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE=debian 2023-07-05T15:02:25.607 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://www.ubuntu.com/" 2023-07-05T15:02:25.607 INFO:teuthology.orchestra.run.smithi089.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-07-05T15:02:25.607 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-07-05T15:02:25.607 INFO:teuthology.orchestra.run.smithi089.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-07-05T15:02:25.607 INFO:teuthology.orchestra.run.smithi089.stdout:UBUNTU_CODENAME=jammy 2023-07-05T15:02:25.608 INFO:teuthology.provision.fog.smithi089:Deploy complete! 2023-07-05T15:02:25.610 DEBUG:teuthology.parallel:result is None 2023-07-05T15:02:25.772 INFO:teuthology.lock.ops:Checking smithi089.front.sepia.ceph.com 2023-07-05T15:02:25.773 INFO:teuthology.lock.ops:New key found. Updating... 2023-07-05T15:02:25.805 INFO:teuthology.lock.ops:Updating [smithi089.front.sepia.ceph.com]: set os type and version on server 2023-07-05T15:02:25.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-07-05T15:02:25.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-05T15:02:25.877 DEBUG:teuthology.orchestra.run.smithi089:> true 2023-07-05T15:02:25.976 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-07-05T15:02:25.977 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2023-07-05T15:02:26.029 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2023-07-05T15:02:26.029 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2023-07-05T15:02:26.076 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-07-05T15:02:26.077 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="Ubuntu" 2023-07-05T15:02:26.077 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="22.04" 2023-07-05T15:02:26.077 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-07-05T15:02:26.077 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_CODENAME=jammy 2023-07-05T15:02:26.077 INFO:teuthology.orchestra.run.smithi089.stdout:ID=ubuntu 2023-07-05T15:02:26.078 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE=debian 2023-07-05T15:02:26.078 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://www.ubuntu.com/" 2023-07-05T15:02:26.078 INFO:teuthology.orchestra.run.smithi089.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-07-05T15:02:26.078 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-07-05T15:02:26.078 INFO:teuthology.orchestra.run.smithi089.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-07-05T15:02:26.078 INFO:teuthology.orchestra.run.smithi089.stdout:UBUNTU_CODENAME=jammy 2023-07-05T15:02:26.079 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2023-07-05T15:02:26.103 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:02:26.104 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:02:26.208 INFO:teuthology.dispatcher.supervisor:Running job 7327287 2023-07-05T15:02:26.208 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/virtualenv/bin/teuthology -v --owner scheduled_jcollin@teuthology --archive /home/teuthworker/archive/jcollin-2023-07-05_14:46:41-fs:mirror-ha-wip-pdonnell-testing-20230705.003205-distro-default-smithi/7327287 --name jcollin-2023-07-05_14:46:41-fs:mirror-ha-wip-pdonnell-testing-20230705.003205-distro-default-smithi --description fs:mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} -- /home/teuthworker/archive/jcollin-2023-07-05_14:46:41-fs:mirror-ha-wip-pdonnell-testing-20230705.003205-distro-default-smithi/7327287/orig.config.yaml 2023-07-05T15:02:26.212 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/jcollin-2023-07-05_14:46:41-fs:mirror-ha-wip-pdonnell-testing-20230705.003205-distro-default-smithi/7327287 2023-07-05T15:02:26.213 INFO:teuthology.dispatcher.supervisor:Job PID: 31305 2023-07-05T15:02:26.213 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-07-05T15:04:26.312 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:04:26.313 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:06:26.488 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:06:26.489 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:08:26.653 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:08:26.653 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:10:26.821 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:10:26.822 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:12:26.993 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:12:26.993 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:14:27.169 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:14:27.170 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:16:27.345 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:16:27.346 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:18:27.516 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:18:27.517 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:20:27.673 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:20:27.674 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:22:27.849 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:22:27.850 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:24:28.036 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:24:28.037 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:26:28.210 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:26:28.211 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:28:28.381 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:28:28.382 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:30:28.557 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:30:28.558 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:32:28.729 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:32:28.730 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:34:28.893 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-05T15:34:28.894 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-05T15:34:28.968 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120 2023-07-05T15:34:29.055 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-07-05T15:34:29.055 INFO:teuthology.nuke:Checking targets against current locks 2023-07-05T15:34:29.104 DEBUG:teuthology.nuke:shortname: smithi089 2023-07-05T15:34:29.105 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-05T15:34:29.128 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/jcollin-2023-07-05_14:46:41-fs:mirror-ha-wip-pdonnell-testing-20230705.003205-distro-default-smithi/7327287', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-05 14:56:50.956721', 'locked_by': 'scheduled_jcollin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF2Q4kMj6rHVt5SpgYeNruAMXHskrwNzR1v5bYlEuZcd'} 2023-07-05T15:34:29.184 INFO:teuthology.orchestra.console:Power off smithi089 2023-07-05T15:34:29.185 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-05T15:34:29.206 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-05T15:34:29.330 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2023-07-05T15:34:29.483 INFO:teuthology.lock.ops:unlocked: smithi089.front.sepia.ceph.com