2024-03-07T13:37:34.680 INFO:root:teuthology version: 0.0.1.dev262+ge691533
2024-03-07T13:37:34.680 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:37:34.818 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f3537e00f40>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f3538e74c10>, signals=[15])
2024-03-07T13:37:34.818 INFO:teuthology.lock.ops:Start node 'smithi042.front.sepia.ceph.com' reimaging
2024-03-07T13:37:34.818 INFO:teuthology.lock.ops:Updating [smithi042.front.sepia.ceph.com]: reset os type and version on server
2024-03-07T13:37:34.818 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server
2024-03-07T13:37:34.840 INFO:teuthology.lock.ops:Start node 'smithi119.front.sepia.ceph.com' reimaging
2024-03-07T13:37:34.840 INFO:teuthology.lock.ops:Updating [smithi119.front.sepia.ceph.com]: reset os type and version on server
2024-03-07T13:37:34.840 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server
2024-03-07T13:37:34.958 DEBUG:teuthology.provision.fog.smithi042:Requesting image smithi_ubuntu_22.04 (ID 6)
2024-03-07T13:37:34.972 DEBUG:teuthology.provision.fog.smithi119:Requesting image smithi_ubuntu_22.04 (ID 6)
2024-03-07T13:37:35.002 INFO:teuthology.provision.fog.smithi042:Scheduling deploy of ubuntu 22.04
2024-03-07T13:37:35.029 INFO:teuthology.provision.fog.smithi119:Scheduling deploy of ubuntu 22.04
2024-03-07T13:37:35.267 INFO:teuthology.orchestra.console.smithi119:Power off
2024-03-07T13:37:35.267 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-07T13:37:35.291 INFO:teuthology.orchestra.console.smithi042:Power off
2024-03-07T13:37:35.291 DEBUG:teuthology.orchestra.console.smithi042:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-07T13:37:35.299 DEBUG:teuthology.orchestra.console.smithi119:power off output: Chassis Power Control: Down/Off
2024-03-07T13:37:35.299 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:35.313 DEBUG:teuthology.orchestra.console.smithi042:power off output: Chassis Power Control: Down/Off
2024-03-07T13:37:35.313 DEBUG:teuthology.orchestra.console.smithi042:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:35.323 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is off
2024-03-07T13:37:35.335 DEBUG:teuthology.orchestra.console.smithi042:check power output: Chassis Power is off
2024-03-07T13:37:35.424 INFO:teuthology.orchestra.console.smithi119:Power off completed
2024-03-07T13:37:35.436 INFO:teuthology.orchestra.console.smithi042:Power off completed
2024-03-07T13:37:35.525 INFO:teuthology.orchestra.console.smithi119:Power on
2024-03-07T13:37:35.525 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-07T13:37:35.537 INFO:teuthology.orchestra.console.smithi042:Power on
2024-03-07T13:37:35.537 DEBUG:teuthology.orchestra.console.smithi042:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-07T13:37:35.547 DEBUG:teuthology.orchestra.console.smithi119:power on output: Chassis Power Control: Up/On
2024-03-07T13:37:35.547 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:35.561 DEBUG:teuthology.orchestra.console.smithi042:power on output: Chassis Power Control: Up/On
2024-03-07T13:37:35.562 DEBUG:teuthology.orchestra.console.smithi042:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:35.574 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is off
2024-03-07T13:37:35.583 DEBUG:teuthology.orchestra.console.smithi042:check power output: Chassis Power is off
2024-03-07T13:37:39.576 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:39.583 DEBUG:teuthology.orchestra.console.smithi042:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:39.685 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is off
2024-03-07T13:37:39.692 DEBUG:teuthology.orchestra.console.smithi042:check power output: Chassis Power is on
2024-03-07T13:37:39.792 INFO:teuthology.orchestra.console.smithi042:Power on completed
2024-03-07T13:37:39.893 INFO:teuthology.provision.fog.smithi042:Waiting for deploy to finish
2024-03-07T13:37:43.689 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-07T13:37:43.798 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on
2024-03-07T13:37:43.898 INFO:teuthology.orchestra.console.smithi119:Power on completed
2024-03-07T13:37:43.999 INFO:teuthology.provision.fog.smithi119:Waiting for deploy to finish
2024-03-07T13:40:56.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:40:59.420 WARNING:teuthology.provision.fog.smithi042:[Errno None] Unable to connect to port 22 on 172.21.15.42
2024-03-07T13:41:00.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:03.292 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2024-03-07T13:41:05.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:08.476 WARNING:teuthology.provision.fog.smithi042:[Errno None] Unable to connect to port 22 on 172.21.15.42
2024-03-07T13:41:09.293 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:12.348 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2024-03-07T13:41:14.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:17.532 WARNING:teuthology.provision.fog.smithi042:[Errno None] Unable to connect to port 22 on 172.21.15.42
2024-03-07T13:41:18.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:21.404 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2024-03-07T13:41:23.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:26.588 WARNING:teuthology.provision.fog.smithi042:[Errno None] Unable to connect to port 22 on 172.21.15.42
2024-03-07T13:41:27.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:30.460 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2024-03-07T13:41:32.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:35.644 WARNING:teuthology.provision.fog.smithi042:[Errno None] Unable to connect to port 22 on 172.21.15.42
2024-03-07T13:41:36.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:39.516 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2024-03-07T13:41:41.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:45.520 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:41:48.576 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2024-03-07T13:41:54.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:42:13.052 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2024-03-07T13:42:19.060 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:42:22.140 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2024-03-07T13:42:28.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:42:28.143 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2024-03-07T13:42:34.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:42:34.152 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2024-03-07T13:42:40.157 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:42:40.159 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2024-03-07T13:42:41.653 WARNING:teuthology.provision.fog.smithi042:timed out
2024-03-07T13:42:46.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:42:46.166 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2024-03-07T13:42:47.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:42:52.172 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:42:52.531 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-07T13:42:52.532 DEBUG:teuthology.orchestra.run.smithi119:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-07T13:42:53.405 INFO:teuthology.provision.fog.smithi119:Node is ready
2024-03-07T13:42:53.405 DEBUG:teuthology.orchestra.run.smithi119:> hostname
2024-03-07T13:42:53.451 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119.front.sepia.ceph.com
2024-03-07T13:42:53.451 DEBUG:teuthology.orchestra.run.smithi119:> grep smithi119.front.sepia.ceph.com /etc/hosts
2024-03-07T13:42:53.499 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119 smithi119.front.sepia.ceph.com smithi119
2024-03-07T13:42:53.499 DEBUG:teuthology.orchestra.run.smithi119:> sudo hostname smithi119
2024-03-07T13:42:53.619 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i -e 's/smithi119.front.sepia.ceph.com/smithi119/g' /etc/hosts
2024-03-07T13:42:53.633 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i -e 's/172.21.15.119/172.21.15.119/g' /etc/hosts
2024-03-07T13:42:53.691 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i -e 's/smithi119.front.sepia.ceph.com/smithi119/g' /etc/hostname
2024-03-07T13:42:53.746 DEBUG:teuthology.orchestra.run.smithi119:> sudo hostname smithi119
2024-03-07T13:42:53.801 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release
2024-03-07T13:42:53.846 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2024-03-07T13:42:53.847 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="Ubuntu"
2024-03-07T13:42:53.847 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="22.04"
2024-03-07T13:42:53.847 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2024-03-07T13:42:53.847 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_CODENAME=jammy
2024-03-07T13:42:53.847 INFO:teuthology.orchestra.run.smithi119.stdout:ID=ubuntu
2024-03-07T13:42:53.847 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE=debian
2024-03-07T13:42:53.847 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://www.ubuntu.com/"
2024-03-07T13:42:53.847 INFO:teuthology.orchestra.run.smithi119.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2024-03-07T13:42:53.847 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2024-03-07T13:42:53.847 INFO:teuthology.orchestra.run.smithi119.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2024-03-07T13:42:53.847 INFO:teuthology.orchestra.run.smithi119.stdout:UBUNTU_CODENAME=jammy
2024-03-07T13:42:53.848 INFO:teuthology.provision.fog.smithi119:Deploy complete!
2024-03-07T13:42:53.849 DEBUG:teuthology.parallel:result is None
2024-03-07T13:42:54.780 WARNING:teuthology.provision.fog.smithi042:[Errno None] Unable to connect to port 22 on 172.21.15.42
2024-03-07T13:43:00.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:43:01.044 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-07T13:43:01.044 DEBUG:teuthology.orchestra.run.smithi042:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-07T13:43:01.577 INFO:teuthology.provision.fog.smithi042:Node is ready
2024-03-07T13:43:01.577 DEBUG:teuthology.orchestra.run.smithi042:> hostname
2024-03-07T13:43:01.623 INFO:teuthology.orchestra.run.smithi042.stdout:smithi042.front.sepia.ceph.com
2024-03-07T13:43:01.623 DEBUG:teuthology.orchestra.run.smithi042:> grep smithi042.front.sepia.ceph.com /etc/hosts
2024-03-07T13:43:01.672 INFO:teuthology.orchestra.run.smithi042.stdout:172.21.15.42 smithi042.front.sepia.ceph.com smithi042
2024-03-07T13:43:01.672 DEBUG:teuthology.orchestra.run.smithi042:> sudo hostname smithi042
2024-03-07T13:43:01.832 DEBUG:teuthology.orchestra.run.smithi042:> sudo sed -i -e 's/smithi042.front.sepia.ceph.com/smithi042/g' /etc/hosts
2024-03-07T13:43:01.846 DEBUG:teuthology.orchestra.run.smithi042:> sudo sed -i -e 's/172.21.15.42/172.21.15.42/g' /etc/hosts
2024-03-07T13:43:01.907 DEBUG:teuthology.orchestra.run.smithi042:> sudo sed -i -e 's/smithi042.front.sepia.ceph.com/smithi042/g' /etc/hostname
2024-03-07T13:43:01.963 DEBUG:teuthology.orchestra.run.smithi042:> sudo hostname smithi042
2024-03-07T13:43:02.018 DEBUG:teuthology.orchestra.run.smithi042:> cat /etc/os-release
2024-03-07T13:43:02.071 INFO:teuthology.orchestra.run.smithi042.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2024-03-07T13:43:02.071 INFO:teuthology.orchestra.run.smithi042.stdout:NAME="Ubuntu"
2024-03-07T13:43:02.071 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_ID="22.04"
2024-03-07T13:43:02.071 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2024-03-07T13:43:02.071 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_CODENAME=jammy
2024-03-07T13:43:02.071 INFO:teuthology.orchestra.run.smithi042.stdout:ID=ubuntu
2024-03-07T13:43:02.071 INFO:teuthology.orchestra.run.smithi042.stdout:ID_LIKE=debian
2024-03-07T13:43:02.071 INFO:teuthology.orchestra.run.smithi042.stdout:HOME_URL="https://www.ubuntu.com/"
2024-03-07T13:43:02.071 INFO:teuthology.orchestra.run.smithi042.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2024-03-07T13:43:02.071 INFO:teuthology.orchestra.run.smithi042.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2024-03-07T13:43:02.071 INFO:teuthology.orchestra.run.smithi042.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2024-03-07T13:43:02.071 INFO:teuthology.orchestra.run.smithi042.stdout:UBUNTU_CODENAME=jammy
2024-03-07T13:43:02.072 INFO:teuthology.provision.fog.smithi042:Deploy complete!
2024-03-07T13:43:02.072 DEBUG:teuthology.parallel:result is None
2024-03-07T13:43:02.253 INFO:teuthology.lock.ops:Checking smithi042.front.sepia.ceph.com
2024-03-07T13:43:02.253 INFO:teuthology.lock.ops:New key found. Updating...
2024-03-07T13:43:02.279 INFO:teuthology.lock.ops:Checking smithi119.front.sepia.ceph.com
2024-03-07T13:43:02.279 INFO:teuthology.lock.ops:New key found. Updating...
2024-03-07T13:43:02.305 INFO:teuthology.lock.ops:Updating [smithi042.front.sepia.ceph.com]: set os type and version on server
2024-03-07T13:43:02.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com'
2024-03-07T13:43:02.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:43:02.391 DEBUG:teuthology.orchestra.run.smithi042:> true
2024-03-07T13:43:02.480 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com'
2024-03-07T13:43:02.480 DEBUG:teuthology.orchestra.run.smithi042:> uname -m
2024-03-07T13:43:02.526 INFO:teuthology.orchestra.run.smithi042.stdout:x86_64
2024-03-07T13:43:02.526 DEBUG:teuthology.orchestra.run.smithi042:> cat /etc/os-release
2024-03-07T13:43:02.578 INFO:teuthology.orchestra.run.smithi042.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2024-03-07T13:43:02.578 INFO:teuthology.orchestra.run.smithi042.stdout:NAME="Ubuntu"
2024-03-07T13:43:02.578 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_ID="22.04"
2024-03-07T13:43:02.578 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2024-03-07T13:43:02.578 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_CODENAME=jammy
2024-03-07T13:43:02.578 INFO:teuthology.orchestra.run.smithi042.stdout:ID=ubuntu
2024-03-07T13:43:02.578 INFO:teuthology.orchestra.run.smithi042.stdout:ID_LIKE=debian
2024-03-07T13:43:02.579 INFO:teuthology.orchestra.run.smithi042.stdout:HOME_URL="https://www.ubuntu.com/"
2024-03-07T13:43:02.579 INFO:teuthology.orchestra.run.smithi042.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2024-03-07T13:43:02.579 INFO:teuthology.orchestra.run.smithi042.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2024-03-07T13:43:02.579 INFO:teuthology.orchestra.run.smithi042.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2024-03-07T13:43:02.579 INFO:teuthology.orchestra.run.smithi042.stdout:UBUNTU_CODENAME=jammy
2024-03-07T13:43:02.579 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server
2024-03-07T13:43:02.604 INFO:teuthology.lock.ops:Updating [smithi119.front.sepia.ceph.com]: set os type and version on server
2024-03-07T13:43:02.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com'
2024-03-07T13:43:02.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-07T13:43:02.684 DEBUG:teuthology.orchestra.run.smithi119:> true
2024-03-07T13:43:02.792 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com'
2024-03-07T13:43:02.792 DEBUG:teuthology.orchestra.run.smithi119:> uname -m
2024-03-07T13:43:02.838 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64
2024-03-07T13:43:02.838 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release
2024-03-07T13:43:02.886 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2024-03-07T13:43:02.886 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="Ubuntu"
2024-03-07T13:43:02.886 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="22.04"
2024-03-07T13:43:02.886 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2024-03-07T13:43:02.886 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_CODENAME=jammy
2024-03-07T13:43:02.886 INFO:teuthology.orchestra.run.smithi119.stdout:ID=ubuntu
2024-03-07T13:43:02.886 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE=debian
2024-03-07T13:43:02.886 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://www.ubuntu.com/"
2024-03-07T13:43:02.886 INFO:teuthology.orchestra.run.smithi119.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2024-03-07T13:43:02.886 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2024-03-07T13:43:02.887 INFO:teuthology.orchestra.run.smithi119.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2024-03-07T13:43:02.887 INFO:teuthology.orchestra.run.smithi119.stdout:UBUNTU_CODENAME=jammy
2024-03-07T13:43:02.887 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server
2024-03-07T13:43:02.912 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:43:03.011 INFO:teuthology.dispatcher.supervisor:Running job 7584482
2024-03-07T13:43:03.011 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7584482 --name vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} -- /home/teuthworker/archive/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7584482/orig.config.yaml
2024-03-07T13:43:03.015 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7584482
2024-03-07T13:43:03.015 INFO:teuthology.dispatcher.supervisor:Job PID: 694251
2024-03-07T13:43:03.016 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2024-03-07T13:45:03.117 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:47:03.289 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:49:03.467 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T13:50:03.535 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 504: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:50:59.614 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:02.529 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:06.397 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:11.219 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:16.993 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:23.723 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:31.484 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:40.124 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:49.720 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

2024-03-07T13:51:49.721 ERROR:teuthology.dispatcher.supervisor:Failed to report job status; ignoring
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 335, in run_with_watchdog
    report.try_push_job_info(job_info)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/report.py", line 512, in try_push_job_info
    push_job_info(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/report.py", line 474, in push_job_info
    reporter.report_job(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/report.py", line 299, in report_job
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'report job 7584482' reached maximum tries (11) after waiting for 52.9115124395632 seconds
2024-03-07T13:53:49.818 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:10:18.447 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:12:18.532 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:14:18.708 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:16:18.858 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:18:19.044 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:20:19.219 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:22:19.395 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:24:19.575 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:26:19.747 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:28:19.917 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:30:20.087 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:32:20.246 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:34:20.415 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:36:20.587 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:38:20.763 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:40:20.947 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:42:21.119 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:44:21.289 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:46:21.456 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:48:21.592 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:50:21.763 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:52:21.955 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:54:22.135 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:56:22.299 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T19:58:22.475 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:00:22.649 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:02:22.819 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:04:22.995 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:06:23.215 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:08:23.402 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:10:23.575 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:12:23.735 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:14:23.911 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:16:24.080 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:18:24.271 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:20:24.442 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:22:24.609 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:24:24.783 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:26:24.955 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:28:25.112 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:30:25.282 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:32:25.453 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:34:25.623 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:36:25.791 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:38:25.979 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:40:26.145 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:42:26.318 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:44:26.481 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:46:26.783 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:48:26.967 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:50:27.155 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:52:27.339 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:54:27.502 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:56:27.674 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T20:58:27.840 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:00:28.017 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:02:28.190 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:04:28.355 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:06:28.517 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:08:28.691 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:10:28.866 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:12:29.033 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:14:29.211 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:16:29.395 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:18:29.563 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:20:29.755 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:22:29.933 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:24:30.095 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:26:30.257 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:28:30.423 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:30:30.599 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:32:30.766 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:34:30.943 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:36:31.109 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:38:31.283 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:40:31.455 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:42:31.623 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:44:31.795 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:46:31.967 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:48:32.135 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:50:32.313 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:52:32.483 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:54:32.651 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:56:32.805 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T21:58:32.976 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:00:33.151 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:02:33.319 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:04:33.488 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:06:33.659 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:08:33.825 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:10:34.005 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:12:34.179 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:14:34.351 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:16:34.523 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:18:34.697 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:20:34.865 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:22:35.039 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:24:35.223 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:26:35.384 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:28:35.555 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:30:35.731 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:32:35.903 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:34:36.075 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:36:36.243 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:38:36.409 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:40:36.583 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:42:36.763 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:44:36.940 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:46:37.115 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:48:37.295 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:50:37.475 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:52:37.651 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:54:37.827 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:56:37.995 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T22:58:38.163 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:00:38.339 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:02:38.475 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:04:38.648 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:06:38.823 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:08:38.977 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:10:39.148 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:12:39.319 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:14:39.491 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:16:39.658 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:18:39.830 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:20:39.999 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:22:40.169 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:24:40.347 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:26:40.518 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:28:40.659 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:30:40.837 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:32:41.011 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:34:41.171 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:36:41.335 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:38:41.511 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:40:41.691 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:42:41.867 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:44:42.039 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:46:42.201 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:48:42.375 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:50:42.551 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:52:42.715 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:54:42.891 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:56:43.065 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-07T23:58:43.231 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:00:43.395 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:02:43.566 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:04:43.731 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:06:43.889 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:08:44.051 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:10:44.184 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:12:44.351 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:14:44.534 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:16:44.698 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:18:44.868 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:20:45.032 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:22:45.179 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:24:45.353 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:26:45.523 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:28:45.695 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:30:45.861 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:32:46.027 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:34:46.180 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:36:46.355 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:38:46.531 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:40:46.731 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:42:46.899 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:44:47.063 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:46:47.227 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:48:47.411 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:50:47.591 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:52:47.771 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:54:47.947 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:56:48.118 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T00:58:48.307 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:00:48.487 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:02:48.671 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:04:48.775 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:06:48.951 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:08:49.127 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:10:49.300 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:12:49.470 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:14:49.623 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:16:49.795 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:18:49.936 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:20:50.102 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:22:50.243 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:24:50.403 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:26:50.563 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:28:50.735 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:30:50.857 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:32:50.987 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:34:51.119 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:36:51.275 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:38:51.375 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:40:51.542 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:42:51.770 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:44:52.034 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2024-03-08T01:44:52.169 INFO:teuthology.kill:Killing Pids: {694251}
2024-03-08T01:44:53.187 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2024-03-08T01:44:53.187 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2024-03-08T01:44:53.187 INFO:teuthology.misc:Compressing logs...
2024-03-08T01:44:53.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com'
2024-03-08T01:44:53.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-08T01:44:53.273 DEBUG:teuthology.orchestra.run.smithi042:> true
2024-03-08T01:44:53.393 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com'
2024-03-08T01:44:53.393 DEBUG:teuthology.orchestra.run.smithi042:> sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2024-03-08T01:44:53.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com'
2024-03-08T01:44:53.437 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-08T01:44:53.505 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log
2024-03-08T01:44:53.505 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
2024-03-08T01:44:53.505 INFO:teuthology.orchestra.run.smithi042.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log:	/home/ubuntu/cephtest/archive/syslog/kern.log:	 60.8% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2024-03-08T01:44:53.505 INFO:teuthology.orchestra.run.smithi042.stderr: 78.6% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2024-03-08T01:44:53.518 DEBUG:teuthology.orchestra.run.smithi119:> true
2024-03-08T01:44:53.629 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com'
2024-03-08T01:44:53.629 DEBUG:teuthology.orchestra.run.smithi119:> sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2024-03-08T01:44:53.825 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log
2024-03-08T01:44:53.826 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
2024-03-08T01:44:53.826 INFO:teuthology.orchestra.run.smithi119.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log:	/home/ubuntu/cephtest/archive/syslog/kern.log:	  0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2024-03-08T01:44:53.834 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2024-03-08T01:44:53.834 INFO:teuthology.orchestra.run.smithi119.stderr: 77.2% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2024-03-08T01:44:53.839 DEBUG:teuthology.misc:Transferring archived files from smithi042:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7584482/remote/smithi042/
2024-03-08T01:44:53.839 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- .
2024-03-08T01:44:53.866 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7584482/remote/smithi119/
2024-03-08T01:44:53.866 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- .
2024-03-08T01:44:53.949 INFO:teuthology.kill:No teuthology processes running
2024-03-08T01:44:54.869 INFO:teuthology.kill:Nuking machines: ['smithi042', 'smithi119']
2024-03-08T01:44:55.651 INFO:teuthology.kill:2024-03-08 01:44:55,651.651 INFO:teuthology.nuke:targets:
2024-03-08T01:44:55.652 INFO:teuthology.kill:  smithi042.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOsJNAT3wWtaZssgtyP75y099Hm4vz3acrdLAcEi5TT7zr5g4rAwBI4ASYxi7hjnk1+6LPs97HTYI6tYlD8du5I=
2024-03-08T01:44:55.652 INFO:teuthology.kill:  smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBfNUNXUde4Ou9GfNpao/LfRzbVBLor7j6cRxtW44M2PCxS+g9fDp3prUgm6XK1hQor8nJ0lTUsl+kYmDBBWmH8=
2024-03-08T01:44:55.652 INFO:teuthology.kill:2024-03-08 01:44:55,651.651 INFO:teuthology.nuke:Checking targets against current locks
2024-03-08T01:44:55.701 INFO:teuthology.kill:2024-03-08 01:44:55,701.701 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-08T01:44:55.728 INFO:teuthology.kill:2024-03-08 01:44:55,728.728 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-08T01:44:55.781 INFO:teuthology.kill:2024-03-08 01:44:55,781.781 INFO:teuthology.orchestra.console.smithi042:Power off
2024-03-08T01:44:55.793 INFO:teuthology.kill:2024-03-08 01:44:55,793.793 INFO:teuthology.orchestra.console.smithi119:Power off
2024-03-08T01:45:08.275 INFO:teuthology.kill:2024-03-08 01:45:08,275.275 INFO:teuthology.orchestra.console.smithi042:Power off completed
2024-03-08T01:45:08.282 INFO:teuthology.kill:2024-03-08 01:45:08,282.282 INFO:teuthology.orchestra.console.smithi119:Power off completed
2024-03-08T01:45:08.427 INFO:teuthology.kill:2024-03-08 01:45:08,426.426 INFO:teuthology.lock.ops:unlocked: smithi042.front.sepia.ceph.com
2024-03-08T01:45:08.434 INFO:teuthology.kill:2024-03-08 01:45:08,434.434 INFO:teuthology.lock.ops:unlocked: smithi119.front.sepia.ceph.com
2024-03-08T01:45:08.491 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:47:08.662 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-08T01:47:08.752 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15