2024-01-11T14:34:16.762 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-01-11T14:34:16.763 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T14:34:16.869 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f63140c6f40>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f631512cc10>, signals=[15])
2024-01-11T14:34:16.869 INFO:teuthology.lock.ops:Start node 'smithi037.front.sepia.ceph.com' reimaging
2024-01-11T14:34:16.869 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: reset os type and version on server
2024-01-11T14:34:16.869 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server
2024-01-11T14:34:17.011 DEBUG:teuthology.provision.fog.smithi037:Requesting image smithi_centos_9.stream (ID 10)
2024-01-11T14:34:17.143 INFO:teuthology.provision.fog.smithi037:Scheduling deploy of centos 9.stream
2024-01-11T14:34:17.400 INFO:teuthology.orchestra.console.smithi037:Power off
2024-01-11T14:34:17.400 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-11T14:34:17.421 DEBUG:teuthology.orchestra.console.smithi037:power off output: Chassis Power Control: Down/Off
2024-01-11T14:34:17.422 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-11T14:34:17.442 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on
2024-01-11T14:34:21.445 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-11T14:34:21.553 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on
2024-01-11T14:34:25.558 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-11T14:34:25.667 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on
2024-01-11T14:34:29.668 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-11T14:34:29.776 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off
2024-01-11T14:34:29.877 INFO:teuthology.orchestra.console.smithi037:Power off completed
2024-01-11T14:34:29.977 INFO:teuthology.orchestra.console.smithi037:Power on
2024-01-11T14:34:29.978 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-01-11T14:34:30.002 DEBUG:teuthology.orchestra.console.smithi037:power on output: Chassis Power Control: Up/On
2024-01-11T14:34:30.002 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-11T14:34:30.027 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off
2024-01-11T14:34:34.032 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-11T14:34:34.141 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on
2024-01-11T14:34:34.241 INFO:teuthology.orchestra.console.smithi037:Power on completed
2024-01-11T14:34:34.342 INFO:teuthology.provision.fog.smithi037:Waiting for deploy to finish
2024-01-11T14:36:50.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-11T14:36:50.193 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-01-11T14:37:54.196 WARNING:teuthology.provision.fog.smithi037:timed out
2024-01-11T14:38:00.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-11T14:38:18.297 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-01-11T14:38:18.298 DEBUG:teuthology.orchestra.run.smithi037:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-01-11T14:38:39.405 INFO:teuthology.provision.fog.smithi037:Node is ready
2024-01-11T14:38:39.405 DEBUG:teuthology.orchestra.run.smithi037:> hostname
2024-01-11T14:38:39.424 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037.front.sepia.ceph.com
2024-01-11T14:38:39.425 DEBUG:teuthology.orchestra.run.smithi037:> grep smithi037.front.sepia.ceph.com /etc/hosts
2024-01-11T14:38:39.482 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37 smithi037.front.sepia.ceph.com smithi037
2024-01-11T14:38:39.483 DEBUG:teuthology.orchestra.run.smithi037:> sudo hostname smithi037
2024-01-11T14:38:39.630 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i -e 's/smithi037.front.sepia.ceph.com/smithi037/g' /etc/hosts
2024-01-11T14:38:39.665 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i -e 's/172.21.15.37/172.21.15.37/g' /etc/hosts
2024-01-11T14:38:39.740 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i -e 's/smithi037.front.sepia.ceph.com/smithi037/g' /etc/hostname
2024-01-11T14:38:39.813 DEBUG:teuthology.orchestra.run.smithi037:> sudo hostname smithi037
2024-01-11T14:38:39.885 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release
2024-01-11T14:38:39.940 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="CentOS Stream"
2024-01-11T14:38:39.940 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="9"
2024-01-11T14:38:39.940 INFO:teuthology.orchestra.run.smithi037.stdout:ID="centos"
2024-01-11T14:38:39.940 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="rhel fedora"
2024-01-11T14:38:39.940 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="9"
2024-01-11T14:38:39.940 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el9"
2024-01-11T14:38:39.941 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="CentOS Stream 9"
2024-01-11T14:38:39.941 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31"
2024-01-11T14:38:39.941 INFO:teuthology.orchestra.run.smithi037.stdout:LOGO="fedora-logo-icon"
2024-01-11T14:38:39.941 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-01-11T14:38:39.941 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://centos.org/"
2024-01-11T14:38:39.941 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-01-11T14:38:39.941 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-01-11T14:38:39.941 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-01-11T14:38:39.942 INFO:teuthology.provision.fog.smithi037:Deploy complete!
2024-01-11T14:38:39.943 DEBUG:teuthology.parallel:result is None
2024-01-11T14:38:40.102 INFO:teuthology.lock.ops:Checking smithi037.front.sepia.ceph.com
2024-01-11T14:38:40.103 INFO:teuthology.lock.ops:New key found. Updating...
2024-01-11T14:38:40.127 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: set os type and version on server
2024-01-11T14:38:40.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2024-01-11T14:38:40.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-11T14:38:40.228 DEBUG:teuthology.orchestra.run.smithi037:> true
2024-01-11T14:38:40.295 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com'
2024-01-11T14:38:40.295 DEBUG:teuthology.orchestra.run.smithi037:> uname -m
2024-01-11T14:38:40.350 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64
2024-01-11T14:38:40.351 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release
2024-01-11T14:38:40.405 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="CentOS Stream"
2024-01-11T14:38:40.405 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="9"
2024-01-11T14:38:40.405 INFO:teuthology.orchestra.run.smithi037.stdout:ID="centos"
2024-01-11T14:38:40.406 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="rhel fedora"
2024-01-11T14:38:40.406 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="9"
2024-01-11T14:38:40.406 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el9"
2024-01-11T14:38:40.406 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="CentOS Stream 9"
2024-01-11T14:38:40.406 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31"
2024-01-11T14:38:40.406 INFO:teuthology.orchestra.run.smithi037.stdout:LOGO="fedora-logo-icon"
2024-01-11T14:38:40.406 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-01-11T14:38:40.406 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://centos.org/"
2024-01-11T14:38:40.406 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-01-11T14:38:40.406 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-01-11T14:38:40.406 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-01-11T14:38:40.407 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server
2024-01-11T14:38:40.426 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T14:38:40.522 INFO:teuthology.dispatcher.supervisor:Running job 7511393
2024-01-11T14:38:40.522 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2024-01-10_15:00:23-fs-wip-vshankar-testing-20240103.072409-1-testing-default-smithi/7511393 --name vshankar-2024-01-10_15:00:23-fs-wip-vshankar-testing-20240103.072409-1-testing-default-smithi --description fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} -- /home/teuthworker/archive/vshankar-2024-01-10_15:00:23-fs-wip-vshankar-testing-20240103.072409-1-testing-default-smithi/7511393/orig.config.yaml
2024-01-11T14:38:40.525 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2024-01-10_15:00:23-fs-wip-vshankar-testing-20240103.072409-1-testing-default-smithi/7511393
2024-01-11T14:38:40.526 INFO:teuthology.dispatcher.supervisor:Job PID: 2577795
2024-01-11T14:38:40.526 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2024-01-11T14:40:40.527 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T14:42:40.595 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T14:44:40.671 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T14:46:40.736 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T14:48:40.809 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T14:50:40.890 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T14:52:40.955 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T14:54:41.038 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T14:56:41.116 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T14:58:41.186 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:00:41.252 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:02:41.318 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:04:41.385 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:06:41.477 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:08:41.541 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:10:41.606 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:12:41.675 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:14:41.748 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:16:41.815 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:18:41.889 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:20:41.955 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:22:42.059 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:24:42.135 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:26:42.213 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:28:42.286 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:30:42.359 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:32:42.433 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:34:42.511 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:36:42.585 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:38:42.657 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:40:42.737 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:42:42.803 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:44:42.867 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:46:42.945 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:48:43.008 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:50:43.083 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:52:43.160 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:54:43.305 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:56:43.362 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T15:58:43.435 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T16:00:43.507 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T16:02:43.580 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T16:04:43.648 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T16:06:43.730 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T16:08:43.803 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T16:10:43.879 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T16:12:43.969 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-11T16:12:44.039 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1
2024-01-11T16:12:44.183 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2024-01-11T16:12:44.183 INFO:teuthology.nuke:Checking targets against current locks
2024-01-11T16:12:44.228 DEBUG:teuthology.nuke:shortname: smithi037
2024-01-11T16:12:44.228 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-01-11T16:12:44.252 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-10_15:00:23-fs-wip-vshankar-testing-20240103.072409-1-testing-default-smithi/7511393', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-11 14:34:15.774690', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDDKJ7p/JuLN2nRBv8QE0JWyOfbOE6Fx9e4NQSiOO1+47MIYHYh9M4yxY/aSsRMhiNmNr/M/30WED+H33+fn+Zk='}
2024-01-11T16:12:44.298 INFO:teuthology.orchestra.console.smithi037:Power off
2024-01-11T16:12:44.298 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-11T16:12:44.319 DEBUG:teuthology.orchestra.console.smithi037:power off output: Chassis Power Control: Down/Off
2024-01-11T16:12:44.319 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-11T16:12:44.340 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off
2024-01-11T16:12:44.441 INFO:teuthology.orchestra.console.smithi037:Power off completed
2024-01-11T16:12:44.588 INFO:teuthology.lock.ops:unlocked: smithi037.front.sepia.ceph.com