2021-08-13T17:53:44.778 INFO:root:teuthology version: 1.1.0-2b512d98 2021-08-13T17:53:44.899 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi026.front.sepia.ceph.com' reimaging 2021-08-13T17:53:44.900 INFO:teuthology.lock.ops:Updating [ubuntu@smithi026.front.sepia.ceph.com]: reset os type and version on server 2021-08-13T17:53:44.901 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2021-08-13T17:53:44.916 INFO:teuthology.lock.ops:Node 'ubuntu@smithi026.front.sepia.ceph.com' reimaging is complete 2021-08-13T17:53:44.916 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi184.front.sepia.ceph.com' reimaging 2021-08-13T17:53:44.917 INFO:teuthology.lock.ops:Updating [ubuntu@smithi184.front.sepia.ceph.com]: reset os type and version on server 2021-08-13T17:53:44.917 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2021-08-13T17:53:44.936 INFO:teuthology.lock.ops:Node 'ubuntu@smithi184.front.sepia.ceph.com' reimaging is complete 2021-08-13T17:53:45.119 INFO:teuthology.provision.fog.smithi026:Scheduling deploy of centos 8.3 2021-08-13T17:53:45.146 INFO:teuthology.provision.fog.smithi184:Scheduling deploy of centos 8.3 2021-08-13T17:53:45.534 INFO:teuthology.orchestra.console:Power off smithi026 2021-08-13T17:53:46.612 INFO:teuthology.orchestra.console:Power off smithi184 2021-08-13T17:53:54.979 INFO:teuthology.orchestra.console:Power off for smithi184 completed 2021-08-13T17:53:55.080 INFO:teuthology.orchestra.console:Power on smithi184 2021-08-13T17:53:58.016 INFO:teuthology.orchestra.console:Power off for smithi026 completed 2021-08-13T17:53:58.118 INFO:teuthology.orchestra.console:Power on smithi026 2021-08-13T17:54:03.444 INFO:teuthology.orchestra.console:Power on for smithi184 completed 2021-08-13T17:54:03.545 INFO:teuthology.provision.fog.smithi184:Waiting for deploy to finish 2021-08-13T17:54:06.489 INFO:teuthology.orchestra.console:Power on for smithi026 completed 2021-08-13T17:54:06.590 INFO:teuthology.provision.fog.smithi026:Waiting for deploy to finish 2021-08-13T17:57:52.005 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-13T17:58:02.213 INFO:teuthology.provision.fog.smithi026:Node is ready 2021-08-13T17:58:02.233 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026.front.sepia.ceph.com 2021-08-13T17:58:02.292 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26 smithi026.front.sepia.ceph.com smithi026 2021-08-13T17:58:02.648 INFO:teuthology.provision.fog.smithi026:Deploy complete! 2021-08-13T17:58:36.509 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-13T17:58:36.802 INFO:teuthology.provision.fog.smithi184:Node is ready 2021-08-13T17:58:36.857 INFO:teuthology.orchestra.run.smithi184.stdout:smithi184.front.sepia.ceph.com 2021-08-13T17:58:36.913 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.15.184 smithi184.front.sepia.ceph.com smithi184 2021-08-13T17:58:37.229 INFO:teuthology.provision.fog.smithi184:Deploy complete! 2021-08-13T17:58:37.380 INFO:teuthology.lock.ops:Checking smithi026.front.sepia.ceph.com 2021-08-13T17:58:37.381 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-13T17:58:37.398 INFO:teuthology.lock.ops:Checking smithi184.front.sepia.ceph.com 2021-08-13T17:58:37.398 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-13T17:58:37.413 INFO:teuthology.lock.ops:Updating [smithi026.front.sepia.ceph.com]: set os type and version on server 2021-08-13T17:58:37.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-13T17:58:37.706 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2021-08-13T17:58:37.761 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="CentOS Linux" 2021-08-13T17:58:37.762 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="8" 2021-08-13T17:58:37.762 INFO:teuthology.orchestra.run.smithi026.stdout:ID="centos" 2021-08-13T17:58:37.763 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="rhel fedora" 2021-08-13T17:58:37.763 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="8" 2021-08-13T17:58:37.763 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el8" 2021-08-13T17:58:37.763 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="CentOS Linux 8" 2021-08-13T17:58:37.764 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2021-08-13T17:58:37.764 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-13T17:58:37.764 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://centos.org/" 2021-08-13T17:58:37.764 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-08-13T17:58:37.764 INFO:teuthology.orchestra.run.smithi026.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-08-13T17:58:37.765 INFO:teuthology.orchestra.run.smithi026.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-08-13T17:58:37.766 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2021-08-13T17:58:37.786 INFO:teuthology.lock.ops:Updating [smithi184.front.sepia.ceph.com]: set os type and version on server 2021-08-13T17:58:37.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-13T17:58:38.093 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2021-08-13T17:58:38.148 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="CentOS Linux" 2021-08-13T17:58:38.148 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="8" 2021-08-13T17:58:38.148 INFO:teuthology.orchestra.run.smithi184.stdout:ID="centos" 2021-08-13T17:58:38.148 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE="rhel fedora" 2021-08-13T17:58:38.149 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="8" 2021-08-13T17:58:38.149 INFO:teuthology.orchestra.run.smithi184.stdout:PLATFORM_ID="platform:el8" 2021-08-13T17:58:38.149 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="CentOS Linux 8" 2021-08-13T17:58:38.149 INFO:teuthology.orchestra.run.smithi184.stdout:ANSI_COLOR="0;31" 2021-08-13T17:58:38.150 INFO:teuthology.orchestra.run.smithi184.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-13T17:58:38.150 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://centos.org/" 2021-08-13T17:58:38.150 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-08-13T17:58:38.152 INFO:teuthology.orchestra.run.smithi184.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-08-13T17:58:38.152 INFO:teuthology.orchestra.run.smithi184.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-08-13T17:58:38.153 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2021-08-13T17:58:38.256 INFO:teuthology.dispatcher.supervisor:Running job 6320871 2021-08-13T17:58:38.257 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_2b512d9834652f3d2dcf3654362107f85adcd013/virtualenv/bin/teuthology -v --owner scheduled_khiremat@teuthology --archive /home/teuthworker/archive/khiremat-2021-08-06_03:55:32-fs-wip-khiremat-test-kernel-exclude-failures-distro-basic-smithi/6320871 --name khiremat-2021-08-06_03:55:32-fs-wip-khiremat-test-kernel-exclude-failures-distro-basic-smithi --description fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays} -- /home/teuthworker/archive/khiremat-2021-08-06_03:55:32-fs-wip-khiremat-test-kernel-exclude-failures-distro-basic-smithi/6320871/orig.config.yaml 2021-08-13T17:58:38.262 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/khiremat-2021-08-06_03:55:32-fs-wip-khiremat-test-kernel-exclude-failures-distro-basic-smithi/6320871 2021-08-13T17:58:38.263 INFO:teuthology.dispatcher.supervisor:Job PID: 25696 2021-08-13T17:58:38.264 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-08-14T00:05:05.356 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/khiremat-2021-08-06_03:55:32-fs-wip-khiremat-test-kernel-exclude-failures-distro-basic-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2021-08-14T05:59:33.631 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-08-14T05:59:33.908 INFO:teuthology.kill:Killing Pids: {25696} 2021-08-14T05:59:35.157 INFO:teuthology.kill:Nuking machines: ['smithi026', 'smithi184'] 2021-08-14T05:59:36.395 INFO:teuthology.kill:2021-08-14 05:59:36,394.394 INFO:teuthology.nuke:targets: 2021-08-14T05:59:36.395 INFO:teuthology.kill: smithi026.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVGpgelWFO9hexgCHJI1qpQRRW7O8a7wiHQwzXxfmIl66lPZnFONf5h7rJ3HIs8JfjRm16En4jktN5o6L0a9VmSn3zq97L6y408EJm0I4y7UeN+mBbRmUB20SWGgpvuYWhdthb6BMO7dqLyv4+CHs9u0pE5PeukpNeTFKi9Ixe12DjZQzpv7/aC5N+XeazZ61Q5AD1DqXl8/Daa87pBusfnTz42MrB0gChT+7M5PolbKC/25lSv3nFWDjeWP6AXgBTvKdZujL7AfPEZebo3fOgtVTr8WNs4MgFrrLLe12FJuiucXu/DXAaL8M9EGZ089YaGGOnHYwv2QAOc2Ib1DIN/wG8mgYBtt30mjISiM+/SuhLZ7N/dlTIZHauJjAPjCQkdb85XmFFKyCR8zMo9M142EqRUsEqN+hn8Navy7egP2ZIjg+pyk3jvBQn7LUmvCNVFuCU4oKyvUczcWEUbnP50lo0YqbcMTp9CxSSXNJaHMRO3sCp8GY335wxGAfKFrk= 2021-08-14T05:59:36.396 INFO:teuthology.kill: smithi184.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/Ke4KEa5bhLkIJG9wk3M7k+T0u/PXl4gNpCFuV9yHnjbiJ1nN1vD4lAvvdgODjGZpoqgUVxV74VP0aJXWoQs6DyYs4MOPOVKB9OQx4nSUa0rSloC6fZBbjzK8K9gLORhXeOfTghTkZZcZaXzU2cg8A8FCwzAO+eG9VHxNXxNS3u8e3/KXgIFcm1mKCVEyJ0vet/M8/DDEqmxw57loIRHWQVCNQ48Z5jh0b/iAB0KPyCQND8jbcTVh+bkjRFLo777SCoCuQ1XQm86S+azP0xDfgZdb2YV/uwtK1k25E+w3AgOh3ubydlEOBI0u0ZiMhtJUo78js46OIVFnHivSUjm74qPlBoRmnujBX2g756DjJzvspfzMhXuRFdFWNGyONDu8hQ/4AO/oYXffsBYDnQIl7LZrFabupQpX3kKXb2IBH5W+DSoZ+W2N0g4vVxrPX3vJRdQL5o9hWQN2OrbFZgLUttMpu6xyRMij1oWeS60ahHkg9ArK0pBh4frgdoWNbp8= 2021-08-14T05:59:36.396 INFO:teuthology.kill:2021-08-14 05:59:36,395.395 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-14T05:59:36.397 INFO:teuthology.kill:2021-08-14 05:59:36,397.397 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-14T05:59:36.508 INFO:teuthology.kill:2021-08-14 05:59:36,508.508 INFO:teuthology.orchestra.console:Power off smithi184 2021-08-14T05:59:36.534 INFO:teuthology.kill:2021-08-14 05:59:36,534.534 INFO:teuthology.orchestra.console:Power off smithi026 2021-08-14T05:59:44.977 INFO:teuthology.kill:2021-08-14 05:59:44,976.976 INFO:teuthology.orchestra.console:Power off for smithi026 completed 2021-08-14T05:59:45.067 INFO:teuthology.kill:2021-08-14 05:59:45,067.067 INFO:teuthology.orchestra.console:Power off for smithi184 completed 2021-08-14T05:59:45.322 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-08-14T05:59:45.323 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-08-14T05:59:45.323 INFO:teuthology.misc:Compressing logs... 2021-08-14T05:59:45.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-14T06:00:45.387 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_2b512d9834652f3d2dcf3654362107f85adcd013/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2b512d9834652f3d2dcf3654362107f85adcd013/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2b512d9834652f3d2dcf3654362107f85adcd013/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_2b512d9834652f3d2dcf3654362107f85adcd013/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_2b512d9834652f3d2dcf3654362107f85adcd013/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_2b512d9834652f3d2dcf3654362107f85adcd013/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2b512d9834652f3d2dcf3654362107f85adcd013/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-08-14T06:00:45.455 INFO:teuthology.kill:No teuthology processes running 2021-08-14T06:00:46.677 INFO:teuthology.kill:Nuking machines: ['smithi026', 'smithi184'] 2021-08-14T06:00:47.686 INFO:teuthology.kill:2021-08-14 06:00:47,686.686 INFO:teuthology.nuke:targets: 2021-08-14T06:00:47.688 INFO:teuthology.kill: smithi026.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVGpgelWFO9hexgCHJI1qpQRRW7O8a7wiHQwzXxfmIl66lPZnFONf5h7rJ3HIs8JfjRm16En4jktN5o6L0a9VmSn3zq97L6y408EJm0I4y7UeN+mBbRmUB20SWGgpvuYWhdthb6BMO7dqLyv4+CHs9u0pE5PeukpNeTFKi9Ixe12DjZQzpv7/aC5N+XeazZ61Q5AD1DqXl8/Daa87pBusfnTz42MrB0gChT+7M5PolbKC/25lSv3nFWDjeWP6AXgBTvKdZujL7AfPEZebo3fOgtVTr8WNs4MgFrrLLe12FJuiucXu/DXAaL8M9EGZ089YaGGOnHYwv2QAOc2Ib1DIN/wG8mgYBtt30mjISiM+/SuhLZ7N/dlTIZHauJjAPjCQkdb85XmFFKyCR8zMo9M142EqRUsEqN+hn8Navy7egP2ZIjg+pyk3jvBQn7LUmvCNVFuCU4oKyvUczcWEUbnP50lo0YqbcMTp9CxSSXNJaHMRO3sCp8GY335wxGAfKFrk= 2021-08-14T06:00:47.688 INFO:teuthology.kill: smithi184.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/Ke4KEa5bhLkIJG9wk3M7k+T0u/PXl4gNpCFuV9yHnjbiJ1nN1vD4lAvvdgODjGZpoqgUVxV74VP0aJXWoQs6DyYs4MOPOVKB9OQx4nSUa0rSloC6fZBbjzK8K9gLORhXeOfTghTkZZcZaXzU2cg8A8FCwzAO+eG9VHxNXxNS3u8e3/KXgIFcm1mKCVEyJ0vet/M8/DDEqmxw57loIRHWQVCNQ48Z5jh0b/iAB0KPyCQND8jbcTVh+bkjRFLo777SCoCuQ1XQm86S+azP0xDfgZdb2YV/uwtK1k25E+w3AgOh3ubydlEOBI0u0ZiMhtJUo78js46OIVFnHivSUjm74qPlBoRmnujBX2g756DjJzvspfzMhXuRFdFWNGyONDu8hQ/4AO/oYXffsBYDnQIl7LZrFabupQpX3kKXb2IBH5W+DSoZ+W2N0g4vVxrPX3vJRdQL5o9hWQN2OrbFZgLUttMpu6xyRMij1oWeS60ahHkg9ArK0pBh4frgdoWNbp8= 2021-08-14T06:00:47.707 INFO:teuthology.kill:2021-08-14 06:00:47,707.707 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-14T06:00:47.710 INFO:teuthology.kill:2021-08-14 06:00:47,710.710 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-14T06:00:47.773 INFO:teuthology.kill:2021-08-14 06:00:47,772.772 INFO:teuthology.orchestra.console:Power off smithi184 2021-08-14T06:00:47.792 INFO:teuthology.kill:2021-08-14 06:00:47,792.792 INFO:teuthology.orchestra.console:Power off smithi026 2021-08-14T06:00:47.928 INFO:teuthology.kill:2021-08-14 06:00:47,928.928 INFO:teuthology.orchestra.console:Power off for smithi184 completed 2021-08-14T06:00:47.935 INFO:teuthology.kill:2021-08-14 06:00:47,934.934 INFO:teuthology.orchestra.console:Power off for smithi026 completed 2021-08-14T06:00:48.078 INFO:teuthology.kill:2021-08-14 06:00:48,077.077 INFO:teuthology.lock.ops:unlocked smithi026.front.sepia.ceph.com 2021-08-14T06:00:48.082 INFO:teuthology.kill:2021-08-14 06:00:48,081.081 INFO:teuthology.lock.ops:unlocked smithi184.front.sepia.ceph.com 2021-08-14T06:02:48.495 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15