2021-09-16T04:18:55.836 INFO:root:teuthology version: 1.1.0-b135a4f8 2021-09-16T04:18:55.925 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi060.front.sepia.ceph.com' reimaging 2021-09-16T04:18:55.927 INFO:teuthology.lock.ops:Updating [ubuntu@smithi060.front.sepia.ceph.com]: reset os type and version on server 2021-09-16T04:18:55.927 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2021-09-16T04:18:55.944 INFO:teuthology.lock.ops:Node 'ubuntu@smithi060.front.sepia.ceph.com' reimaging is complete 2021-09-16T04:18:55.945 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi193.front.sepia.ceph.com' reimaging 2021-09-16T04:18:55.945 INFO:teuthology.lock.ops:Updating [ubuntu@smithi193.front.sepia.ceph.com]: reset os type and version on server 2021-09-16T04:18:55.946 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2021-09-16T04:18:55.964 INFO:teuthology.lock.ops:Node 'ubuntu@smithi193.front.sepia.ceph.com' reimaging is complete 2021-09-16T04:18:56.135 INFO:teuthology.provision.fog.smithi060:Scheduling deploy of centos 8.3 2021-09-16T04:18:56.153 INFO:teuthology.provision.fog.smithi193:Scheduling deploy of centos 8.3 2021-09-16T04:18:56.498 INFO:teuthology.orchestra.console:Power off smithi193 2021-09-16T04:18:56.510 INFO:teuthology.orchestra.console:Power off smithi060 2021-09-16T04:19:04.866 INFO:teuthology.orchestra.console:Power off for smithi193 completed 2021-09-16T04:19:04.871 INFO:teuthology.orchestra.console:Power off for smithi060 completed 2021-09-16T04:19:04.968 INFO:teuthology.orchestra.console:Power on smithi193 2021-09-16T04:19:04.975 INFO:teuthology.orchestra.console:Power on smithi060 2021-09-16T04:19:13.332 INFO:teuthology.orchestra.console:Power on for smithi193 completed 2021-09-16T04:19:13.338 INFO:teuthology.orchestra.console:Power on for smithi060 completed 2021-09-16T04:19:13.434 INFO:teuthology.provision.fog.smithi193:Waiting for deploy to finish 2021-09-16T04:19:13.439 INFO:teuthology.provision.fog.smithi060:Waiting for deploy to finish 2021-09-16T04:23:20.988 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-16T04:23:21.303 INFO:teuthology.provision.fog.smithi060:Node is ready 2021-09-16T04:23:21.318 INFO:teuthology.orchestra.run.smithi060.stdout:smithi060.front.sepia.ceph.com 2021-09-16T04:23:21.373 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60 smithi060.front.sepia.ceph.com smithi060 2021-09-16T04:23:21.688 INFO:teuthology.provision.fog.smithi060:Deploy complete! 2021-09-16T04:23:45.183 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-16T04:23:45.479 INFO:teuthology.provision.fog.smithi193:Node is ready 2021-09-16T04:23:45.494 INFO:teuthology.orchestra.run.smithi193.stdout:smithi193.front.sepia.ceph.com 2021-09-16T04:23:45.550 INFO:teuthology.orchestra.run.smithi193.stdout:172.21.15.193 smithi193.front.sepia.ceph.com smithi193 2021-09-16T04:23:45.873 INFO:teuthology.provision.fog.smithi193:Deploy complete! 2021-09-16T04:23:46.023 INFO:teuthology.lock.ops:Checking smithi060.front.sepia.ceph.com 2021-09-16T04:23:46.024 INFO:teuthology.lock.ops:New key found. Updating... 2021-09-16T04:23:46.049 INFO:teuthology.lock.ops:Checking smithi193.front.sepia.ceph.com 2021-09-16T04:23:46.049 INFO:teuthology.lock.ops:New key found. Updating... 2021-09-16T04:23:46.064 INFO:teuthology.lock.ops:Updating [smithi060.front.sepia.ceph.com]: set os type and version on server 2021-09-16T04:23:46.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-16T04:23:46.337 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2021-09-16T04:23:46.392 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="CentOS Linux" 2021-09-16T04:23:46.393 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="8" 2021-09-16T04:23:46.393 INFO:teuthology.orchestra.run.smithi060.stdout:ID="centos" 2021-09-16T04:23:46.393 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="rhel fedora" 2021-09-16T04:23:46.393 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="8" 2021-09-16T04:23:46.394 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el8" 2021-09-16T04:23:46.394 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="CentOS Linux 8" 2021-09-16T04:23:46.394 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2021-09-16T04:23:46.394 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-09-16T04:23:46.395 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://centos.org/" 2021-09-16T04:23:46.395 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-09-16T04:23:46.395 INFO:teuthology.orchestra.run.smithi060.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-09-16T04:23:46.395 INFO:teuthology.orchestra.run.smithi060.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-09-16T04:23:46.397 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2021-09-16T04:23:46.418 INFO:teuthology.lock.ops:Updating [smithi193.front.sepia.ceph.com]: set os type and version on server 2021-09-16T04:23:46.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-16T04:23:46.674 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2021-09-16T04:23:46.728 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Linux" 2021-09-16T04:23:46.728 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="8" 2021-09-16T04:23:46.728 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2021-09-16T04:23:46.729 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2021-09-16T04:23:46.729 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="8" 2021-09-16T04:23:46.729 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el8" 2021-09-16T04:23:46.729 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Linux 8" 2021-09-16T04:23:46.729 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2021-09-16T04:23:46.730 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-09-16T04:23:46.730 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2021-09-16T04:23:46.730 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-09-16T04:23:46.730 INFO:teuthology.orchestra.run.smithi193.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-09-16T04:23:46.730 INFO:teuthology.orchestra.run.smithi193.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-09-16T04:23:46.731 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2021-09-16T04:23:46.810 INFO:teuthology.dispatcher.supervisor:Running job 6391544 2021-09-16T04:23:46.810 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-09-15_21:19:39-rados-wip-yuri2-testing-2021-09-15-1233-distro-basic-smithi/6391544 --name yuriw-2021-09-15_21:19:39-rados-wip-yuri2-testing-2021-09-15-1233-distro-basic-smithi --description rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} -- /home/teuthworker/archive/yuriw-2021-09-15_21:19:39-rados-wip-yuri2-testing-2021-09-15-1233-distro-basic-smithi/6391544/orig.config.yaml 2021-09-16T04:23:46.814 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-09-15_21:19:39-rados-wip-yuri2-testing-2021-09-15-1233-distro-basic-smithi/6391544 2021-09-16T04:23:46.815 INFO:teuthology.dispatcher.supervisor:Job PID: 29762 2021-09-16T04:23:46.815 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-09-16T11:06:16.189 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2021-09-15_21:19:39-rados-wip-yuri2-testing-2021-09-15-1233-distro-basic-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2021-09-16T16:24:38.147 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-09-16T16:24:38.307 INFO:teuthology.kill:Killing Pids: {29762} 2021-09-16T16:24:39.463 INFO:teuthology.kill:Nuking machines: ['smithi060', 'smithi193'] 2021-09-16T16:24:40.396 INFO:teuthology.kill:2021-09-16 16:24:40,396.396 INFO:teuthology.nuke:targets: 2021-09-16T16:24:40.397 INFO:teuthology.kill: smithi060.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLeFE5ccVDNMMa8Xaz3GaEgK6TTVBLG1Gqwae7m8OH4wS5RHLLshkiW713QBLQwdL1sMRdr0bGOK4iyD18CGZtODe0bo7dSvEBPf9fG7ANkuPlP7xykkOvIxsrXlHmz5UzMQHw0Lwv/04BkgqVXXGEEYhz9xTWJ89gie/GaS4BvM6JgDSCnj77MW/YKoC3bPHnEi58GS0zisdcCpnBxgj8USYIHJrURNbIXYzQd4CiZEplqpO3qnWzkI6siMad9iPWCpFM2s6x25Bw6IKUmTBkAcq0RSWOeSqhsdk7MPgtqniDIIT2sLpwZs06vYuz5nY9+8Py0zRpXKFlvJNVs74rKclAhI4+bTgAPai4UwUWTzkmZV4LoZPY1xh02hE+8r/qVEWi//hP/J/DwN/77h8BeiKHI1noayocubsvvjqBC5QH8gOWgi1zh02ReRChSIoeFpMGMST55UtgVkF5Ajvf/YGSEApWZF5QN7NgNM8lMZiHEZNin6JBd2CxJRxA9xs= 2021-09-16T16:24:40.397 INFO:teuthology.kill: smithi193.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUEqqKqA/dn60LtllIp2lX7jvMoqCmgwNqbj8xVU/Fs8bICAg6kq1d0b25Ol2A7IoL/iiF5W46pzrgdvhLi+hjk+XjvNj5Svx+fuiJda0B+ju20zLV8/jjtdrJnX9ZL05OqAX7Zl86bKl8k6cwcXW6Tvg1q05OTscmyaH4uxhsNjoz0hQwBZD4vzzsxpBjQvTqv5bvymBERBv8gXllrvH4NrkC23bmvN3N/ownXftNYDERpHv5Q95dbt/WBJPm99W4+oGzHQPp4Bb+Qnh3anVmplTcQNnmhHXF7kszd0LvHpNHhO2KsaKCHj1gS5nDjHc2lxazA+Q6XfieJvQv59V2ZDo7HbnmbZz6irRRpvg/u+7HpBg4V8x8OKAqGDWFF1AtjVlDZ0iMFyGf4Z+nI8jwuP7posZ7ZqindJfTjMrBlVQJG2qYGNSBb7G0SjO+lmXt7VlnxGcMAhwFu9WumC2yz4u53mGIE+/xh4pPwljGAEJEbR3gVjqKLAkMIqfzyKc= 2021-09-16T16:24:40.397 INFO:teuthology.kill:2021-09-16 16:24:40,396.396 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-16T16:24:40.399 INFO:teuthology.kill:2021-09-16 16:24:40,398.398 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-16T16:24:40.454 INFO:teuthology.kill:2021-09-16 16:24:40,454.454 INFO:teuthology.orchestra.console:Power off smithi060 2021-09-16T16:24:40.467 INFO:teuthology.kill:2021-09-16 16:24:40,467.467 INFO:teuthology.orchestra.console:Power off smithi193 2021-09-16T16:24:48.828 INFO:teuthology.kill:2021-09-16 16:24:48,828.828 INFO:teuthology.orchestra.console:Power off for smithi060 completed 2021-09-16T16:24:48.835 INFO:teuthology.kill:2021-09-16 16:24:48,835.835 INFO:teuthology.orchestra.console:Power off for smithi193 completed 2021-09-16T16:24:49.086 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2'] 2021-09-16T16:24:49.087 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['osd.3', 'osd.4', 'osd.5', 'client.0'] 2021-09-16T16:24:49.089 INFO:teuthology.misc:Compressing logs... 2021-09-16T16:24:49.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-16T16:25:49.148 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-09-16T16:25:49.211 INFO:teuthology.kill:No teuthology processes running 2021-09-16T16:25:50.357 INFO:teuthology.kill:Nuking machines: ['smithi060', 'smithi193'] 2021-09-16T16:25:51.366 INFO:teuthology.kill:2021-09-16 16:25:51,366.366 INFO:teuthology.nuke:targets: 2021-09-16T16:25:51.367 INFO:teuthology.kill: smithi060.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLeFE5ccVDNMMa8Xaz3GaEgK6TTVBLG1Gqwae7m8OH4wS5RHLLshkiW713QBLQwdL1sMRdr0bGOK4iyD18CGZtODe0bo7dSvEBPf9fG7ANkuPlP7xykkOvIxsrXlHmz5UzMQHw0Lwv/04BkgqVXXGEEYhz9xTWJ89gie/GaS4BvM6JgDSCnj77MW/YKoC3bPHnEi58GS0zisdcCpnBxgj8USYIHJrURNbIXYzQd4CiZEplqpO3qnWzkI6siMad9iPWCpFM2s6x25Bw6IKUmTBkAcq0RSWOeSqhsdk7MPgtqniDIIT2sLpwZs06vYuz5nY9+8Py0zRpXKFlvJNVs74rKclAhI4+bTgAPai4UwUWTzkmZV4LoZPY1xh02hE+8r/qVEWi//hP/J/DwN/77h8BeiKHI1noayocubsvvjqBC5QH8gOWgi1zh02ReRChSIoeFpMGMST55UtgVkF5Ajvf/YGSEApWZF5QN7NgNM8lMZiHEZNin6JBd2CxJRxA9xs= 2021-09-16T16:25:51.368 INFO:teuthology.kill: smithi193.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUEqqKqA/dn60LtllIp2lX7jvMoqCmgwNqbj8xVU/Fs8bICAg6kq1d0b25Ol2A7IoL/iiF5W46pzrgdvhLi+hjk+XjvNj5Svx+fuiJda0B+ju20zLV8/jjtdrJnX9ZL05OqAX7Zl86bKl8k6cwcXW6Tvg1q05OTscmyaH4uxhsNjoz0hQwBZD4vzzsxpBjQvTqv5bvymBERBv8gXllrvH4NrkC23bmvN3N/ownXftNYDERpHv5Q95dbt/WBJPm99W4+oGzHQPp4Bb+Qnh3anVmplTcQNnmhHXF7kszd0LvHpNHhO2KsaKCHj1gS5nDjHc2lxazA+Q6XfieJvQv59V2ZDo7HbnmbZz6irRRpvg/u+7HpBg4V8x8OKAqGDWFF1AtjVlDZ0iMFyGf4Z+nI8jwuP7posZ7ZqindJfTjMrBlVQJG2qYGNSBb7G0SjO+lmXt7VlnxGcMAhwFu9WumC2yz4u53mGIE+/xh4pPwljGAEJEbR3gVjqKLAkMIqfzyKc= 2021-09-16T16:25:51.387 INFO:teuthology.kill:2021-09-16 16:25:51,387.387 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-16T16:25:51.390 INFO:teuthology.kill:2021-09-16 16:25:51,389.389 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-16T16:25:51.447 INFO:teuthology.kill:2021-09-16 16:25:51,447.447 INFO:teuthology.orchestra.console:Power off smithi060 2021-09-16T16:25:51.462 INFO:teuthology.kill:2021-09-16 16:25:51,461.461 INFO:teuthology.orchestra.console:Power off smithi193 2021-09-16T16:25:51.590 INFO:teuthology.kill:2021-09-16 16:25:51,590.590 INFO:teuthology.orchestra.console:Power off for smithi060 completed 2021-09-16T16:25:51.602 INFO:teuthology.kill:2021-09-16 16:25:51,602.602 INFO:teuthology.orchestra.console:Power off for smithi193 completed 2021-09-16T16:25:51.738 INFO:teuthology.kill:2021-09-16 16:25:51,738.738 INFO:teuthology.lock.ops:unlocked smithi060.front.sepia.ceph.com 2021-09-16T16:25:51.750 INFO:teuthology.kill:2021-09-16 16:25:51,750.750 INFO:teuthology.lock.ops:unlocked smithi193.front.sepia.ceph.com 2021-09-16T16:27:52.059 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-09-16T16:27:52.146 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-09-16T16:27:52.146 INFO:teuthology.nuke:Checking targets against current locks 2021-09-16T16:27:52.237 INFO:teuthology.nuke:Not nuking smithi193.front.sepia.ceph.com because description doesn't match 2021-09-16T16:27:52.237 INFO:teuthology.nuke:Not nuking smithi060.front.sepia.ceph.com because description doesn't match