2021-05-26T06:14:17.714 INFO:root:teuthology version: 1.1.0-0c70841a 2021-05-26T06:14:17.817 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi013.front.sepia.ceph.com' reimaging 2021-05-26T06:14:17.818 INFO:teuthology.lock.ops:Updating [ubuntu@smithi013.front.sepia.ceph.com]: reset os type and version on server 2021-05-26T06:14:17.819 INFO:teuthology.lock.ops:Updating smithi013.front.sepia.ceph.com on lock server 2021-05-26T06:14:17.832 INFO:teuthology.lock.ops:Node 'ubuntu@smithi013.front.sepia.ceph.com' reimaging is complete 2021-05-26T06:14:17.832 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi066.front.sepia.ceph.com' reimaging 2021-05-26T06:14:17.833 INFO:teuthology.lock.ops:Updating [ubuntu@smithi066.front.sepia.ceph.com]: reset os type and version on server 2021-05-26T06:14:17.833 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2021-05-26T06:14:17.849 INFO:teuthology.lock.ops:Node 'ubuntu@smithi066.front.sepia.ceph.com' reimaging is complete 2021-05-26T06:14:18.035 INFO:teuthology.provision.fog.smithi013:Scheduling deploy of centos 8.2 2021-05-26T06:14:18.045 INFO:teuthology.provision.fog.smithi066:Scheduling deploy of centos 8.2 2021-05-26T06:14:18.382 INFO:teuthology.orchestra.console:Power off smithi013 2021-05-26T06:14:18.397 INFO:teuthology.orchestra.console:Power off smithi066 2021-05-26T06:14:26.779 INFO:teuthology.orchestra.console:Power off for smithi066 completed 2021-05-26T06:14:26.880 INFO:teuthology.orchestra.console:Power on smithi066 2021-05-26T06:14:30.878 INFO:teuthology.orchestra.console:Power off for smithi013 completed 2021-05-26T06:14:30.980 INFO:teuthology.orchestra.console:Power on smithi013 2021-05-26T06:14:35.233 INFO:teuthology.orchestra.console:Power on for smithi013 completed 2021-05-26T06:14:35.240 INFO:teuthology.orchestra.console:Power on for smithi066 completed 2021-05-26T06:14:35.335 INFO:teuthology.provision.fog.smithi013:Waiting for deploy to finish 2021-05-26T06:14:35.340 INFO:teuthology.provision.fog.smithi066:Waiting for deploy to finish 2021-05-26T06:18:52.020 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-26T06:18:52.332 INFO:teuthology.provision.fog.smithi013:Node is ready 2021-05-26T06:18:52.349 INFO:teuthology.orchestra.run.smithi013.stdout:smithi013.front.sepia.ceph.com 2021-05-26T06:18:52.405 INFO:teuthology.orchestra.run.smithi013.stdout:172.21.15.13 smithi013.front.sepia.ceph.com smithi013 2021-05-26T06:18:52.740 INFO:teuthology.provision.fog.smithi013:Deploy complete! 2021-05-26T06:19:07.019 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-26T06:19:07.317 INFO:teuthology.provision.fog.smithi066:Node is ready 2021-05-26T06:19:07.332 INFO:teuthology.orchestra.run.smithi066.stdout:smithi066.front.sepia.ceph.com 2021-05-26T06:19:07.387 INFO:teuthology.orchestra.run.smithi066.stdout:172.21.15.66 smithi066.front.sepia.ceph.com smithi066 2021-05-26T06:19:07.715 INFO:teuthology.provision.fog.smithi066:Deploy complete! 2021-05-26T06:19:07.845 INFO:teuthology.lock.ops:Checking smithi013.front.sepia.ceph.com 2021-05-26T06:19:07.846 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-26T06:19:07.861 INFO:teuthology.lock.ops:Checking smithi066.front.sepia.ceph.com 2021-05-26T06:19:07.861 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-26T06:19:07.875 INFO:teuthology.lock.ops:Updating [smithi013.front.sepia.ceph.com]: set os type and version on server 2021-05-26T06:19:07.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-26T06:19:08.152 INFO:teuthology.orchestra.run.smithi013.stdout:x86_64 2021-05-26T06:19:08.207 INFO:teuthology.orchestra.run.smithi013.stdout:NAME="CentOS Linux" 2021-05-26T06:19:08.207 INFO:teuthology.orchestra.run.smithi013.stdout:VERSION="8 (Core)" 2021-05-26T06:19:08.208 INFO:teuthology.orchestra.run.smithi013.stdout:ID="centos" 2021-05-26T06:19:08.208 INFO:teuthology.orchestra.run.smithi013.stdout:ID_LIKE="rhel fedora" 2021-05-26T06:19:08.208 INFO:teuthology.orchestra.run.smithi013.stdout:VERSION_ID="8" 2021-05-26T06:19:08.208 INFO:teuthology.orchestra.run.smithi013.stdout:PLATFORM_ID="platform:el8" 2021-05-26T06:19:08.208 INFO:teuthology.orchestra.run.smithi013.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-05-26T06:19:08.209 INFO:teuthology.orchestra.run.smithi013.stdout:ANSI_COLOR="0;31" 2021-05-26T06:19:08.209 INFO:teuthology.orchestra.run.smithi013.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-26T06:19:08.209 INFO:teuthology.orchestra.run.smithi013.stdout:HOME_URL="https://www.centos.org/" 2021-05-26T06:19:08.209 INFO:teuthology.orchestra.run.smithi013.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-26T06:19:08.210 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-05-26T06:19:08.210 INFO:teuthology.orchestra.run.smithi013.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-26T06:19:08.210 INFO:teuthology.orchestra.run.smithi013.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-26T06:19:08.210 INFO:teuthology.orchestra.run.smithi013.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-05-26T06:19:08.210 INFO:teuthology.orchestra.run.smithi013.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-05-26T06:19:08.211 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-05-26T06:19:08.212 INFO:teuthology.lock.ops:Updating smithi013.front.sepia.ceph.com on lock server 2021-05-26T06:19:08.227 INFO:teuthology.lock.ops:Updating [smithi066.front.sepia.ceph.com]: set os type and version on server 2021-05-26T06:19:08.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-26T06:19:08.498 INFO:teuthology.orchestra.run.smithi066.stdout:x86_64 2021-05-26T06:19:08.554 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="CentOS Linux" 2021-05-26T06:19:08.554 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="8 (Core)" 2021-05-26T06:19:08.554 INFO:teuthology.orchestra.run.smithi066.stdout:ID="centos" 2021-05-26T06:19:08.555 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE="rhel fedora" 2021-05-26T06:19:08.555 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="8" 2021-05-26T06:19:08.555 INFO:teuthology.orchestra.run.smithi066.stdout:PLATFORM_ID="platform:el8" 2021-05-26T06:19:08.555 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-05-26T06:19:08.556 INFO:teuthology.orchestra.run.smithi066.stdout:ANSI_COLOR="0;31" 2021-05-26T06:19:08.556 INFO:teuthology.orchestra.run.smithi066.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-26T06:19:08.556 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://www.centos.org/" 2021-05-26T06:19:08.557 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-26T06:19:08.557 INFO:teuthology.orchestra.run.smithi066.stdout: 2021-05-26T06:19:08.557 INFO:teuthology.orchestra.run.smithi066.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-26T06:19:08.558 INFO:teuthology.orchestra.run.smithi066.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-26T06:19:08.558 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-05-26T06:19:08.558 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-05-26T06:19:08.560 INFO:teuthology.orchestra.run.smithi066.stdout: 2021-05-26T06:19:08.560 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2021-05-26T06:19:08.635 INFO:teuthology.dispatcher.supervisor:Running job 6134878 2021-05-26T06:19:08.635 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-05-25_19:30:22-fs-wip-yuri4-testing-2021-05-25-0940-pacific-distro-basic-smithi/6134878 --name yuriw-2021-05-25_19:30:22-fs-wip-yuri4-testing-2021-05-25-0940-pacific-distro-basic-smithi --description fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{frag_enable osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} -- /home/teuthworker/archive/yuriw-2021-05-25_19:30:22-fs-wip-yuri4-testing-2021-05-25-0940-pacific-distro-basic-smithi/6134878/orig.config.yaml 2021-05-26T06:19:08.639 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-05-25_19:30:22-fs-wip-yuri4-testing-2021-05-25-0940-pacific-distro-basic-smithi/6134878 2021-05-26T06:19:08.640 INFO:teuthology.dispatcher.supervisor:Job PID: 5415 2021-05-26T06:19:08.640 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-26T06:27:09.161 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2021-05-25_19:30:22-fs-wip-yuri4-testing-2021-05-25-0940-pacific-distro-basic-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2021-05-26T18:19:58.595 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-26T18:19:58.808 INFO:teuthology.kill:Killing Pids: {5415} 2021-05-26T18:19:59.962 INFO:teuthology.kill:Nuking machines: ['smithi013', 'smithi066'] 2021-05-26T18:20:00.908 INFO:teuthology.kill:2021-05-26 18:20:00,907.907 INFO:teuthology.nuke:targets: 2021-05-26T18:20:00.909 INFO:teuthology.kill: smithi013.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqcA9BJvM7m/MNVmJ9Dgc1wFZvppTT9jAIhxjXb83P1gS649ahUpQXl6/qEhpDS11MMKaPjQXY/Hr2WhsHsbzqBRH3w/KSk8KIqaoQZ0E4VmI1Br8hnhEoAzve/uG844M1n9FgyZZP++flBMK3odL2pjzFGN3kaHKNEfoW5vzo0kpAen0HFl41MSBlEuk44KwhlB2xv9PJK1tH9pyHQJ5KIgaxIZLUxz8Xrcc09KieBl8frPERzU2xNCQcHBZRBpX4m0Ws9U6giDXop2HEDHkFJVc26sD0NAK2lEJ5vu5/KgJLqUgbrr4g/Nk+Cbkj/uxYm8Dd/S7cX+fF/MDX1ZeTrfdfUzjA0KM2O+QsZrfEezRZ6itd4cFDSjsWv8Bq7jRQcuPCfuHTAElvAI8eDnpNOu0EIt7K4c7ud+9IE4IbcpJMBZapGzuGDiCURpht28xQKRW4oOyHAzNkx4AifSYBT3XCgVULolaHGOckp4GrAuoa9zsjTVM3M/t7PLhUfM= 2021-05-26T18:20:00.909 INFO:teuthology.kill: smithi066.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwfVU51vCPvIfUWl2D61/YQWljDrSVld5pmvRKsaIpB0fvk6T9VgGtZ93GODFHLjHVFwbpzkuWdkh9Kb6P83dUhQ1OurV9wM/a/7SYi00nWyKJsuiSvdLZLeAqzBT01lJicJogH2saB2appbHRykL10M/qitqLElRi+nSABuYKZV6MwbxThGAo0u1Hl2fUPYWma60lHXkttfAIRVUojjABPfunF23Az2bR16M1nNpUEHxp7aaytlUED6caMBBnJC+LNpg2GBZgvErdqQL4Lez/qPPyxq3VhlLteACs85qmORe7Vrd+nLnfTd8RYLnHFtpKnRjtB9bjXakIyp+4FmMdetYVbKmWUMG6x8ekHIoTAqNjs6MhwmpxgYwnaOUOF6cfo2kNZGVVZoXMVqBb3/oAoG5I/VV8Rt9Btd3O2x5TNEACTblvXY8Yc/kxMB/s6eJYbfbOE7eQRNOmj2kPVY2JxJgGSWpQpvHO0QccMVoB6adsSHHrz+eiHNA9VPQxwtU= 2021-05-26T18:20:00.909 INFO:teuthology.kill:2021-05-26 18:20:00,908.908 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T18:20:00.910 INFO:teuthology.kill:2021-05-26 18:20:00,910.910 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T18:20:00.971 INFO:teuthology.kill:2021-05-26 18:20:00,971.971 INFO:teuthology.orchestra.console:Power off smithi066 2021-05-26T18:20:00.984 INFO:teuthology.kill:2021-05-26 18:20:00,983.983 INFO:teuthology.orchestra.console:Power off smithi013 2021-05-26T18:20:09.340 INFO:teuthology.kill:2021-05-26 18:20:09,340.340 INFO:teuthology.orchestra.console:Power off for smithi066 completed 2021-05-26T18:20:09.347 INFO:teuthology.kill:2021-05-26 18:20:09,347.347 INFO:teuthology.orchestra.console:Power off for smithi013 completed 2021-05-26T18:20:09.586 INFO:teuthology.task.internal:roles: ubuntu@smithi013.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-05-26T18:20:09.587 INFO:teuthology.task.internal:roles: ubuntu@smithi066.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2021-05-26T18:20:09.587 INFO:teuthology.misc:Compressing logs... 2021-05-26T18:20:09.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-26T18:21:09.653 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-26T18:21:09.776 INFO:teuthology.kill:No teuthology processes running 2021-05-26T18:21:11.071 INFO:teuthology.kill:Nuking machines: ['smithi013', 'smithi066'] 2021-05-26T18:21:12.024 INFO:teuthology.kill:2021-05-26 18:21:12,023.023 INFO:teuthology.nuke:targets: 2021-05-26T18:21:12.026 INFO:teuthology.kill: smithi013.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqcA9BJvM7m/MNVmJ9Dgc1wFZvppTT9jAIhxjXb83P1gS649ahUpQXl6/qEhpDS11MMKaPjQXY/Hr2WhsHsbzqBRH3w/KSk8KIqaoQZ0E4VmI1Br8hnhEoAzve/uG844M1n9FgyZZP++flBMK3odL2pjzFGN3kaHKNEfoW5vzo0kpAen0HFl41MSBlEuk44KwhlB2xv9PJK1tH9pyHQJ5KIgaxIZLUxz8Xrcc09KieBl8frPERzU2xNCQcHBZRBpX4m0Ws9U6giDXop2HEDHkFJVc26sD0NAK2lEJ5vu5/KgJLqUgbrr4g/Nk+Cbkj/uxYm8Dd/S7cX+fF/MDX1ZeTrfdfUzjA0KM2O+QsZrfEezRZ6itd4cFDSjsWv8Bq7jRQcuPCfuHTAElvAI8eDnpNOu0EIt7K4c7ud+9IE4IbcpJMBZapGzuGDiCURpht28xQKRW4oOyHAzNkx4AifSYBT3XCgVULolaHGOckp4GrAuoa9zsjTVM3M/t7PLhUfM= 2021-05-26T18:21:12.027 INFO:teuthology.kill: smithi066.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwfVU51vCPvIfUWl2D61/YQWljDrSVld5pmvRKsaIpB0fvk6T9VgGtZ93GODFHLjHVFwbpzkuWdkh9Kb6P83dUhQ1OurV9wM/a/7SYi00nWyKJsuiSvdLZLeAqzBT01lJicJogH2saB2appbHRykL10M/qitqLElRi+nSABuYKZV6MwbxThGAo0u1Hl2fUPYWma60lHXkttfAIRVUojjABPfunF23Az2bR16M1nNpUEHxp7aaytlUED6caMBBnJC+LNpg2GBZgvErdqQL4Lez/qPPyxq3VhlLteACs85qmORe7Vrd+nLnfTd8RYLnHFtpKnRjtB9bjXakIyp+4FmMdetYVbKmWUMG6x8ekHIoTAqNjs6MhwmpxgYwnaOUOF6cfo2kNZGVVZoXMVqBb3/oAoG5I/VV8Rt9Btd3O2x5TNEACTblvXY8Yc/kxMB/s6eJYbfbOE7eQRNOmj2kPVY2JxJgGSWpQpvHO0QccMVoB6adsSHHrz+eiHNA9VPQxwtU= 2021-05-26T18:21:12.044 INFO:teuthology.kill:2021-05-26 18:21:12,044.044 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T18:21:12.047 INFO:teuthology.kill:2021-05-26 18:21:12,047.047 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T18:21:12.110 INFO:teuthology.kill:2021-05-26 18:21:12,110.110 INFO:teuthology.orchestra.console:Power off smithi013 2021-05-26T18:21:12.132 INFO:teuthology.kill:2021-05-26 18:21:12,131.131 INFO:teuthology.orchestra.console:Power off smithi066 2021-05-26T18:21:12.266 INFO:teuthology.kill:2021-05-26 18:21:12,266.266 INFO:teuthology.orchestra.console:Power off for smithi013 completed 2021-05-26T18:21:12.280 INFO:teuthology.kill:2021-05-26 18:21:12,278.278 INFO:teuthology.orchestra.console:Power off for smithi066 completed 2021-05-26T18:21:12.413 INFO:teuthology.kill:2021-05-26 18:21:12,412.412 INFO:teuthology.lock.ops:unlocked smithi013.front.sepia.ceph.com 2021-05-26T18:21:12.429 INFO:teuthology.kill:2021-05-26 18:21:12,429.429 INFO:teuthology.lock.ops:unlocked smithi066.front.sepia.ceph.com 2021-05-26T18:23:12.722 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-05-26T18:23:12.814 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-05-26T18:23:12.815 INFO:teuthology.nuke:Checking targets against current locks 2021-05-26T18:23:12.901 INFO:teuthology.nuke:Not nuking smithi066.front.sepia.ceph.com because description doesn't match 2021-05-26T18:23:12.901 INFO:teuthology.nuke:Not nuking smithi013.front.sepia.ceph.com because description doesn't match