2021-06-01T05:27:19.906 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-01T05:27:20.008 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi072.front.sepia.ceph.com' reimaging 2021-06-01T05:27:20.010 INFO:teuthology.lock.ops:Updating [ubuntu@smithi072.front.sepia.ceph.com]: reset os type and version on server 2021-06-01T05:27:20.010 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2021-06-01T05:27:20.027 INFO:teuthology.lock.ops:Node 'ubuntu@smithi072.front.sepia.ceph.com' reimaging is complete 2021-06-01T05:27:20.028 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi122.front.sepia.ceph.com' reimaging 2021-06-01T05:27:20.028 INFO:teuthology.lock.ops:Updating [ubuntu@smithi122.front.sepia.ceph.com]: reset os type and version on server 2021-06-01T05:27:20.028 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2021-06-01T05:27:20.044 INFO:teuthology.lock.ops:Node 'ubuntu@smithi122.front.sepia.ceph.com' reimaging is complete 2021-06-01T05:27:20.233 INFO:teuthology.provision.fog.smithi072:Scheduling deploy of centos 8.2 2021-06-01T05:27:20.247 INFO:teuthology.provision.fog.smithi122:Scheduling deploy of centos 8.2 2021-06-01T05:27:20.627 INFO:teuthology.orchestra.console:Power off smithi122 2021-06-01T05:27:20.647 INFO:teuthology.orchestra.console:Power off smithi072 2021-06-01T05:27:20.777 INFO:teuthology.orchestra.console:Power off for smithi122 completed 2021-06-01T05:27:20.787 INFO:teuthology.orchestra.console:Power off for smithi072 completed 2021-06-01T05:27:20.879 INFO:teuthology.orchestra.console:Power on smithi122 2021-06-01T05:27:20.889 INFO:teuthology.orchestra.console:Power on smithi072 2021-06-01T05:27:25.244 INFO:teuthology.orchestra.console:Power on for smithi072 completed 2021-06-01T05:27:25.345 INFO:teuthology.provision.fog.smithi072:Waiting for deploy to finish 2021-06-01T05:27:29.349 INFO:teuthology.orchestra.console:Power on for smithi122 completed 2021-06-01T05:27:29.450 INFO:teuthology.provision.fog.smithi122:Waiting for deploy to finish 2021-06-01T05:32:10.166 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-01T05:32:10.459 INFO:teuthology.provision.fog.smithi122:Node is ready 2021-06-01T05:32:10.515 INFO:teuthology.orchestra.run.smithi122.stdout:smithi122.front.sepia.ceph.com 2021-06-01T05:32:10.570 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.15.122 smithi122.front.sepia.ceph.com smithi122 2021-06-01T05:32:10.889 INFO:teuthology.provision.fog.smithi122:Deploy complete! 2021-06-01T05:36:01.122 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-01T05:36:11.121 INFO:teuthology.provision.fog.smithi072:Node is ready 2021-06-01T05:36:11.223 INFO:teuthology.orchestra.run.smithi072.stdout:smithi072.front.sepia.ceph.com 2021-06-01T05:36:11.248 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72 smithi072.front.sepia.ceph.com smithi072 2021-06-01T05:36:11.756 INFO:teuthology.provision.fog.smithi072:Deploy complete! 2021-06-01T05:36:11.875 INFO:teuthology.lock.ops:Checking smithi072.front.sepia.ceph.com 2021-06-01T05:36:11.876 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-01T05:36:11.892 INFO:teuthology.lock.ops:Checking smithi122.front.sepia.ceph.com 2021-06-01T05:36:11.892 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-01T05:36:11.908 INFO:teuthology.lock.ops:Updating [smithi072.front.sepia.ceph.com]: set os type and version on server 2021-06-01T05:36:11.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-01T05:36:12.176 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2021-06-01T05:36:12.230 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Linux" 2021-06-01T05:36:12.231 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="8 (Core)" 2021-06-01T05:36:12.231 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos" 2021-06-01T05:36:12.232 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora" 2021-06-01T05:36:12.232 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="8" 2021-06-01T05:36:12.233 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el8" 2021-06-01T05:36:12.233 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-06-01T05:36:12.234 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2021-06-01T05:36:12.234 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-01T05:36:12.235 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://www.centos.org/" 2021-06-01T05:36:12.235 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-01T05:36:12.236 INFO:teuthology.orchestra.run.smithi072.stdout: 2021-06-01T05:36:12.236 INFO:teuthology.orchestra.run.smithi072.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-01T05:36:12.237 INFO:teuthology.orchestra.run.smithi072.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-01T05:36:12.237 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-06-01T05:36:12.237 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-06-01T05:36:12.238 INFO:teuthology.orchestra.run.smithi072.stdout: 2021-06-01T05:36:12.240 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2021-06-01T05:36:12.258 INFO:teuthology.lock.ops:Updating [smithi122.front.sepia.ceph.com]: set os type and version on server 2021-06-01T05:36:12.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-01T05:36:12.531 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2021-06-01T05:36:12.585 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Linux" 2021-06-01T05:36:12.585 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="8 (Core)" 2021-06-01T05:36:12.586 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2021-06-01T05:36:12.586 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2021-06-01T05:36:12.586 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="8" 2021-06-01T05:36:12.586 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el8" 2021-06-01T05:36:12.587 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-06-01T05:36:12.587 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2021-06-01T05:36:12.587 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-01T05:36:12.587 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://www.centos.org/" 2021-06-01T05:36:12.587 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-01T05:36:12.588 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-06-01T05:36:12.588 INFO:teuthology.orchestra.run.smithi122.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-01T05:36:12.588 INFO:teuthology.orchestra.run.smithi122.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-01T05:36:12.588 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-06-01T05:36:12.589 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-06-01T05:36:12.589 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-06-01T05:36:12.589 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2021-06-01T05:36:12.671 INFO:teuthology.dispatcher.supervisor:Running job 6144543 2021-06-01T05:36:12.671 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-06-01_04:17:03-fs-pacific-distro-basic-smithi/6144543 --name teuthology-2021-06-01_04:17:03-fs-pacific-distro-basic-smithi --description fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/fragment} -- /home/teuthworker/archive/teuthology-2021-06-01_04:17:03-fs-pacific-distro-basic-smithi/6144543/orig.config.yaml 2021-06-01T05:36:12.674 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-06-01_04:17:03-fs-pacific-distro-basic-smithi/6144543 2021-06-01T05:36:12.675 INFO:teuthology.dispatcher.supervisor:Job PID: 9620 2021-06-01T05:36:12.675 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-06-01T16:22:55.436 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/teuthology-2021-06-01_04:17:03-fs-pacific-distro-basic-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2021-06-01T17:37:01.400 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-06-01T17:37:01.604 INFO:teuthology.kill:Killing Pids: {9620} 2021-06-01T17:37:02.649 INFO:teuthology.kill:Nuking machines: ['smithi072', 'smithi122'] 2021-06-01T17:37:03.539 INFO:teuthology.kill:2021-06-01 17:37:03,539.539 INFO:teuthology.nuke:targets: 2021-06-01T17:37:03.540 INFO:teuthology.kill: smithi072.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJAfILPfuLdM3iGMhdGk1VqT59ztA3YBNFld8IsEveCqU+0PZ8AykHRvWZi9qtxVJavSuKWGoctjFqePV9AfdC/JaV03d/AGzssYhIEWbxOSStk/e229MejsROmU0iq/rB07/l1WL+1JgSPC260VrXsjLhqsGbbyyo/DsT8Qg0c0L8RTGEiRb/7ZXyNBhtY8PLy9jSFBn1tan9+e9q3EUOqiAgQWrgTlimbRNy7PP1JYoMqcw52mNaG1JzeJFhDVhDNE8bexuSnEiuM2o96WnDxXIbavScJnNhqzGjoL+AEthqY2nxhCAJ76ERq4bOfIyJ/wFbdAYhgCv9tBeu/iGkiOOkGKYQiTo4AufQGo7caBVHi2zPJtQQbiaBGzUtlb7NhNg7oduYFSN9O5mKXiTyOB0yKGqrDGbDsjwjm6qvCkNqlRADvmpmLRPT1qk8C0Y+oTNga/7wbwC0TpHCVC7JqANhXdWhraoQrJBlNzWKBGK0vv2L3HAP5/hcfaSf12c= 2021-06-01T17:37:03.541 INFO:teuthology.kill: smithi122.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUbFROrpSM4319JVbDEYJrQyRTx/9XHT9La1yIV08bdhwl7/MkGbsG4n/TKGn6I/rEmHT66dR+Z2LeZ/KoQPiCbLie96LPtaa9fVkmDq5r4cDpVjj4FYmArr7Th7TySz0q368fDWd62ImzA939939Or++uiM3y5874TCxn7VMry4NC6LU/b4RjOjrc6IISE6cVWPbCbJ8GRPFBGYiTXqM0h/GxhxlDdNnDsBsj2Ceu4POFCtLBP3kx9DJYvrg9n2Oclafe09GhL1KgnGlYw4kHqHI2E2Uiuy8TKvEaMfwMIZu0TEHVpqcZmVUeXXbuFIK4lsmcYHvpbqMQgRe93xctoOVlffQkR/1vvVGQPTJrRZinMzzmWGBJsJtOd+TiI2+kJUlqdrcU4R0InBzzYf1QIezkT1Zz/mnxtusjXOPLsFRkbHesW5Yqjx+URQ6AVB82oRPXAOXOFRoiQ8pX7SmlWhr7YJ9R0hpfOHWwmBOZQYNSS0/Hcile6WvThteXv18= 2021-06-01T17:37:03.541 INFO:teuthology.kill:2021-06-01 17:37:03,539.539 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-01T17:37:03.542 INFO:teuthology.kill:2021-06-01 17:37:03,542.542 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-01T17:37:03.598 INFO:teuthology.kill:2021-06-01 17:37:03,597.597 INFO:teuthology.orchestra.console:Power off smithi122 2021-06-01T17:37:03.612 INFO:teuthology.kill:2021-06-01 17:37:03,612.612 INFO:teuthology.orchestra.console:Power off smithi072 2021-06-01T17:37:11.966 INFO:teuthology.kill:2021-06-01 17:37:11,965.965 INFO:teuthology.orchestra.console:Power off for smithi122 completed 2021-06-01T17:37:16.083 INFO:teuthology.kill:2021-06-01 17:37:16,083.083 INFO:teuthology.orchestra.console:Power off for smithi072 completed 2021-06-01T17:37:16.296 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-06-01T17:37:16.297 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-06-01T17:37:16.297 INFO:teuthology.misc:Compressing logs... 2021-06-01T17:37:16.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-01T17:38:16.352 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-06-01T17:38:16.415 INFO:teuthology.kill:No teuthology processes running 2021-06-01T17:38:17.450 INFO:teuthology.kill:Nuking machines: ['smithi072', 'smithi122'] 2021-06-01T17:38:18.399 INFO:teuthology.kill:2021-06-01 17:38:18,399.399 INFO:teuthology.nuke:targets: 2021-06-01T17:38:18.400 INFO:teuthology.kill: smithi072.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJAfILPfuLdM3iGMhdGk1VqT59ztA3YBNFld8IsEveCqU+0PZ8AykHRvWZi9qtxVJavSuKWGoctjFqePV9AfdC/JaV03d/AGzssYhIEWbxOSStk/e229MejsROmU0iq/rB07/l1WL+1JgSPC260VrXsjLhqsGbbyyo/DsT8Qg0c0L8RTGEiRb/7ZXyNBhtY8PLy9jSFBn1tan9+e9q3EUOqiAgQWrgTlimbRNy7PP1JYoMqcw52mNaG1JzeJFhDVhDNE8bexuSnEiuM2o96WnDxXIbavScJnNhqzGjoL+AEthqY2nxhCAJ76ERq4bOfIyJ/wFbdAYhgCv9tBeu/iGkiOOkGKYQiTo4AufQGo7caBVHi2zPJtQQbiaBGzUtlb7NhNg7oduYFSN9O5mKXiTyOB0yKGqrDGbDsjwjm6qvCkNqlRADvmpmLRPT1qk8C0Y+oTNga/7wbwC0TpHCVC7JqANhXdWhraoQrJBlNzWKBGK0vv2L3HAP5/hcfaSf12c= 2021-06-01T17:38:18.400 INFO:teuthology.kill: smithi122.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUbFROrpSM4319JVbDEYJrQyRTx/9XHT9La1yIV08bdhwl7/MkGbsG4n/TKGn6I/rEmHT66dR+Z2LeZ/KoQPiCbLie96LPtaa9fVkmDq5r4cDpVjj4FYmArr7Th7TySz0q368fDWd62ImzA939939Or++uiM3y5874TCxn7VMry4NC6LU/b4RjOjrc6IISE6cVWPbCbJ8GRPFBGYiTXqM0h/GxhxlDdNnDsBsj2Ceu4POFCtLBP3kx9DJYvrg9n2Oclafe09GhL1KgnGlYw4kHqHI2E2Uiuy8TKvEaMfwMIZu0TEHVpqcZmVUeXXbuFIK4lsmcYHvpbqMQgRe93xctoOVlffQkR/1vvVGQPTJrRZinMzzmWGBJsJtOd+TiI2+kJUlqdrcU4R0InBzzYf1QIezkT1Zz/mnxtusjXOPLsFRkbHesW5Yqjx+URQ6AVB82oRPXAOXOFRoiQ8pX7SmlWhr7YJ9R0hpfOHWwmBOZQYNSS0/Hcile6WvThteXv18= 2021-06-01T17:38:18.420 INFO:teuthology.kill:2021-06-01 17:38:18,420.420 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-01T17:38:18.423 INFO:teuthology.kill:2021-06-01 17:38:18,423.423 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-01T17:38:18.480 INFO:teuthology.kill:2021-06-01 17:38:18,480.480 INFO:teuthology.orchestra.console:Power off smithi122 2021-06-01T17:38:18.494 INFO:teuthology.kill:2021-06-01 17:38:18,493.493 INFO:teuthology.orchestra.console:Power off smithi072 2021-06-01T17:38:18.626 INFO:teuthology.kill:2021-06-01 17:38:18,625.625 INFO:teuthology.orchestra.console:Power off for smithi122 completed 2021-06-01T17:38:18.635 INFO:teuthology.kill:2021-06-01 17:38:18,635.635 INFO:teuthology.orchestra.console:Power off for smithi072 completed 2021-06-01T17:38:18.764 INFO:teuthology.kill:2021-06-01 17:38:18,763.763 INFO:teuthology.lock.ops:unlocked smithi122.front.sepia.ceph.com 2021-06-01T17:38:18.772 INFO:teuthology.kill:2021-06-01 17:38:18,772.772 INFO:teuthology.lock.ops:unlocked smithi072.front.sepia.ceph.com 2021-06-01T17:40:19.049 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-06-01T17:40:19.142 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-06-01T17:40:19.142 INFO:teuthology.nuke:Checking targets against current locks 2021-06-01T17:40:19.218 INFO:teuthology.nuke:Not nuking smithi122.front.sepia.ceph.com because description doesn't match 2021-06-01T17:40:19.218 INFO:teuthology.nuke:Not nuking smithi072.front.sepia.ceph.com because description doesn't match