2021-06-03T14:27:46.886 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-03T14:27:46.987 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi046.front.sepia.ceph.com' reimaging 2021-06-03T14:27:46.988 INFO:teuthology.lock.ops:Updating [ubuntu@smithi046.front.sepia.ceph.com]: reset os type and version on server 2021-06-03T14:27:46.988 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2021-06-03T14:27:47.006 INFO:teuthology.lock.ops:Node 'ubuntu@smithi046.front.sepia.ceph.com' reimaging is complete 2021-06-03T14:27:47.006 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi097.front.sepia.ceph.com' reimaging 2021-06-03T14:27:47.006 INFO:teuthology.lock.ops:Updating [ubuntu@smithi097.front.sepia.ceph.com]: reset os type and version on server 2021-06-03T14:27:47.007 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2021-06-03T14:27:47.023 INFO:teuthology.lock.ops:Node 'ubuntu@smithi097.front.sepia.ceph.com' reimaging is complete 2021-06-03T14:27:47.215 INFO:teuthology.provision.fog.smithi046:Scheduling deploy of centos 8.2 2021-06-03T14:27:47.259 INFO:teuthology.provision.fog.smithi097:Scheduling deploy of centos 8.2 2021-06-03T14:27:47.677 INFO:teuthology.orchestra.console:Power off smithi046 2021-06-03T14:27:47.722 INFO:teuthology.orchestra.console:Power off smithi097 2021-06-03T14:27:56.057 INFO:teuthology.orchestra.console:Power off for smithi046 completed 2021-06-03T14:27:56.158 INFO:teuthology.orchestra.console:Power on smithi046 2021-06-03T14:28:00.199 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2021-06-03T14:28:00.301 INFO:teuthology.orchestra.console:Power on smithi097 2021-06-03T14:28:04.525 INFO:teuthology.orchestra.console:Power on for smithi046 completed 2021-06-03T14:28:04.627 INFO:teuthology.provision.fog.smithi046:Waiting for deploy to finish 2021-06-03T14:28:08.664 INFO:teuthology.orchestra.console:Power on for smithi097 completed 2021-06-03T14:28:08.765 INFO:teuthology.provision.fog.smithi097:Waiting for deploy to finish 2021-06-03T14:32:13.273 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-03T14:32:24.235 INFO:teuthology.provision.fog.smithi046:Node is ready 2021-06-03T14:32:24.254 INFO:teuthology.orchestra.run.smithi046.stdout:smithi046.front.sepia.ceph.com 2021-06-03T14:32:24.312 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46 smithi046.front.sepia.ceph.com smithi046 2021-06-03T14:32:24.663 INFO:teuthology.provision.fog.smithi046:Deploy complete! 2021-06-03T14:32:49.812 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-03T14:32:50.152 INFO:teuthology.provision.fog.smithi097:Node is ready 2021-06-03T14:32:50.168 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097.front.sepia.ceph.com 2021-06-03T14:32:50.223 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97 smithi097.front.sepia.ceph.com smithi097 2021-06-03T14:32:50.557 INFO:teuthology.provision.fog.smithi097:Deploy complete! 2021-06-03T14:32:50.700 INFO:teuthology.lock.ops:Checking smithi046.front.sepia.ceph.com 2021-06-03T14:32:50.700 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-03T14:32:50.716 INFO:teuthology.lock.ops:Checking smithi097.front.sepia.ceph.com 2021-06-03T14:32:50.716 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-03T14:32:50.729 INFO:teuthology.lock.ops:Updating [smithi046.front.sepia.ceph.com]: set os type and version on server 2021-06-03T14:32:50.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-03T14:32:50.948 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2021-06-03T14:32:51.005 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Linux" 2021-06-03T14:32:51.006 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="8 (Core)" 2021-06-03T14:32:51.006 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2021-06-03T14:32:51.006 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2021-06-03T14:32:51.007 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="8" 2021-06-03T14:32:51.007 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el8" 2021-06-03T14:32:51.007 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-06-03T14:32:51.007 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2021-06-03T14:32:51.008 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-03T14:32:51.008 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://www.centos.org/" 2021-06-03T14:32:51.008 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-03T14:32:51.008 INFO:teuthology.orchestra.run.smithi046.stdout: 2021-06-03T14:32:51.008 INFO:teuthology.orchestra.run.smithi046.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-03T14:32:51.009 INFO:teuthology.orchestra.run.smithi046.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-03T14:32:51.009 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-06-03T14:32:51.009 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-06-03T14:32:51.009 INFO:teuthology.orchestra.run.smithi046.stdout: 2021-06-03T14:32:51.011 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2021-06-03T14:32:51.027 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: set os type and version on server 2021-06-03T14:32:51.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-03T14:32:51.290 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2021-06-03T14:32:51.343 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Linux" 2021-06-03T14:32:51.344 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="8 (Core)" 2021-06-03T14:32:51.344 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2021-06-03T14:32:51.345 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2021-06-03T14:32:51.345 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="8" 2021-06-03T14:32:51.345 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el8" 2021-06-03T14:32:51.345 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-06-03T14:32:51.345 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2021-06-03T14:32:51.346 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-03T14:32:51.346 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://www.centos.org/" 2021-06-03T14:32:51.346 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-03T14:32:51.346 INFO:teuthology.orchestra.run.smithi097.stdout: 2021-06-03T14:32:51.347 INFO:teuthology.orchestra.run.smithi097.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-03T14:32:51.347 INFO:teuthology.orchestra.run.smithi097.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-03T14:32:51.347 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-06-03T14:32:51.347 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-06-03T14:32:51.347 INFO:teuthology.orchestra.run.smithi097.stdout: 2021-06-03T14:32:51.348 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2021-06-03T14:32:51.425 INFO:teuthology.dispatcher.supervisor:Running job 6147186 2021-06-03T14:32:51.425 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-06-02_18:29:36-fs-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi/6147186 --name yuriw-2021-06-02_18:29:36-fs-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi --description fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace} -- /home/teuthworker/archive/yuriw-2021-06-02_18:29:36-fs-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi/6147186/orig.config.yaml 2021-06-03T14:32:51.429 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-06-02_18:29:36-fs-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi/6147186 2021-06-03T14:32:51.430 INFO:teuthology.dispatcher.supervisor:Job PID: 7426 2021-06-03T14:32:51.430 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-06-04T00:33:36.439 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2021-06-02_18:29:36-fs-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2021-06-04T02:33:45.857 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-06-04T02:33:46.136 INFO:teuthology.kill:Killing Pids: {7426} 2021-06-04T02:33:47.193 INFO:teuthology.kill:Nuking machines: ['smithi046', 'smithi097'] 2021-06-04T02:33:48.100 INFO:teuthology.kill:2021-06-04 02:33:48,100.100 INFO:teuthology.nuke:targets: 2021-06-04T02:33:48.101 INFO:teuthology.kill: smithi046.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPS61U+hTWkc3zRlgCvyfDllNzHz/yKOWzS7/T6+hmV5aHdTsRtmGDBl2kJrd0sIS8FV1bF8F+yLd45vVhkX9WJbx/SI/j7PLfYCKfeO1EjUxd+0DpbJrTXMuaib/qwEsyrdawgxVw+A/BDCj3PZhQTVAQRwtdX3qASUXWVulaggSJgUACg5YHiwMn9V6euL9AevndCjnPEzofOJQ5XfRUXRu2r4++Z5e2cKtAmrEScrYbCf3GCYAzGFIK+y7E7dCufqpQF4M/uvyMTtGMDvaERea1pOn9wY0GdS+WIn1XVDVmhqUOuojAJ48WLrCvn1BXV2JFlAyB0nKQUelXx+19+uFjQxa9wAXmKZwPSVdRySGZVQ8sfTdgEm7nzl+PjXgCevaTssCnUfuW1vfw0mvmhLEqr3cDsrFNVZS3EMzpyQz+iAbdubGG4JfpQpR4cOq8+FEU0rnBO8CCpBHwO8WY1oFhJ2AvuqotHW4rFDyeWPk0vvZBKi9qIJVTZItym08= 2021-06-04T02:33:48.102 INFO:teuthology.kill: smithi097.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCZ29/o9m8IF7sQZBkXuKQ8bxGHMoNsWfHxat6PsvWtUTIFwhhZqaVWyXHyLI54n7RrpmI4b+sTFH0kDWroTActk6wBDliu5SBH3UNwQzzbN48e2NNnRP0c6it/AmvM3vSI/yNWhc1ECMeW4IMoadwFJYt8JPV5Oje2Ahhfv73/GMy5ACEMBDblj8THPfbxgoxH3zgtEBj/pTG/+QdboVcnAp4AIR3q3KEdSOr6Kxuy9gBYiv0nAxjxVHfnjaQSv+kteNBsVMvGJDOv3NIiqHMmIyA3RCn558ddVEwL1y2JQW08IsKxLVkEydG+25NK/F8BKwQ4zP+hXHRpg+OuCSWnRR0Zm8yVAFHyXFOWlOmoZaUWV8Xzva28dOmaCf6GY1Vc9todTff3yt4wYLEdwOanlHOOqW+GIZYiORTBoJ3FxypoesgGR/pmALMalOFONgkMbTal3wlzv89JMvbLcIiJF1oOG9FPsDrdOLNZqa/ikXvYAzO4db9b8eTMaXo4qSk= 2021-06-04T02:33:48.103 INFO:teuthology.kill:2021-06-04 02:33:48,100.100 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-04T02:33:48.103 INFO:teuthology.kill:2021-06-04 02:33:48,102.102 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-04T02:33:48.167 INFO:teuthology.kill:2021-06-04 02:33:48,166.166 INFO:teuthology.orchestra.console:Power off smithi097 2021-06-04T02:33:48.180 INFO:teuthology.kill:2021-06-04 02:33:48,180.180 INFO:teuthology.orchestra.console:Power off smithi046 2021-06-04T02:33:56.537 INFO:teuthology.kill:2021-06-04 02:33:56,537.537 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2021-06-04T02:33:56.550 INFO:teuthology.kill:2021-06-04 02:33:56,550.550 INFO:teuthology.orchestra.console:Power off for smithi046 completed 2021-06-04T02:33:56.797 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-06-04T02:33:56.797 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-06-04T02:33:56.798 INFO:teuthology.misc:Compressing logs... 2021-06-04T02:33:56.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-04T02:34:56.860 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-06-04T02:34:56.941 INFO:teuthology.kill:No teuthology processes running 2021-06-04T02:34:58.035 INFO:teuthology.kill:Nuking machines: ['smithi046', 'smithi097'] 2021-06-04T02:34:58.970 INFO:teuthology.kill:2021-06-04 02:34:58,970.970 INFO:teuthology.nuke:targets: 2021-06-04T02:34:58.971 INFO:teuthology.kill: smithi046.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPS61U+hTWkc3zRlgCvyfDllNzHz/yKOWzS7/T6+hmV5aHdTsRtmGDBl2kJrd0sIS8FV1bF8F+yLd45vVhkX9WJbx/SI/j7PLfYCKfeO1EjUxd+0DpbJrTXMuaib/qwEsyrdawgxVw+A/BDCj3PZhQTVAQRwtdX3qASUXWVulaggSJgUACg5YHiwMn9V6euL9AevndCjnPEzofOJQ5XfRUXRu2r4++Z5e2cKtAmrEScrYbCf3GCYAzGFIK+y7E7dCufqpQF4M/uvyMTtGMDvaERea1pOn9wY0GdS+WIn1XVDVmhqUOuojAJ48WLrCvn1BXV2JFlAyB0nKQUelXx+19+uFjQxa9wAXmKZwPSVdRySGZVQ8sfTdgEm7nzl+PjXgCevaTssCnUfuW1vfw0mvmhLEqr3cDsrFNVZS3EMzpyQz+iAbdubGG4JfpQpR4cOq8+FEU0rnBO8CCpBHwO8WY1oFhJ2AvuqotHW4rFDyeWPk0vvZBKi9qIJVTZItym08= 2021-06-04T02:34:58.971 INFO:teuthology.kill: smithi097.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCZ29/o9m8IF7sQZBkXuKQ8bxGHMoNsWfHxat6PsvWtUTIFwhhZqaVWyXHyLI54n7RrpmI4b+sTFH0kDWroTActk6wBDliu5SBH3UNwQzzbN48e2NNnRP0c6it/AmvM3vSI/yNWhc1ECMeW4IMoadwFJYt8JPV5Oje2Ahhfv73/GMy5ACEMBDblj8THPfbxgoxH3zgtEBj/pTG/+QdboVcnAp4AIR3q3KEdSOr6Kxuy9gBYiv0nAxjxVHfnjaQSv+kteNBsVMvGJDOv3NIiqHMmIyA3RCn558ddVEwL1y2JQW08IsKxLVkEydG+25NK/F8BKwQ4zP+hXHRpg+OuCSWnRR0Zm8yVAFHyXFOWlOmoZaUWV8Xzva28dOmaCf6GY1Vc9todTff3yt4wYLEdwOanlHOOqW+GIZYiORTBoJ3FxypoesgGR/pmALMalOFONgkMbTal3wlzv89JMvbLcIiJF1oOG9FPsDrdOLNZqa/ikXvYAzO4db9b8eTMaXo4qSk= 2021-06-04T02:34:58.991 INFO:teuthology.kill:2021-06-04 02:34:58,991.991 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-04T02:34:58.995 INFO:teuthology.kill:2021-06-04 02:34:58,995.995 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-04T02:34:59.049 INFO:teuthology.kill:2021-06-04 02:34:59,048.048 INFO:teuthology.orchestra.console:Power off smithi046 2021-06-04T02:34:59.067 INFO:teuthology.kill:2021-06-04 02:34:59,067.067 INFO:teuthology.orchestra.console:Power off smithi097 2021-06-04T02:34:59.216 INFO:teuthology.kill:2021-06-04 02:34:59,216.216 INFO:teuthology.orchestra.console:Power off for smithi046 completed 2021-06-04T02:34:59.226 INFO:teuthology.kill:2021-06-04 02:34:59,226.226 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2021-06-04T02:34:59.355 INFO:teuthology.kill:2021-06-04 02:34:59,354.354 INFO:teuthology.lock.ops:unlocked smithi046.front.sepia.ceph.com 2021-06-04T02:34:59.363 INFO:teuthology.kill:2021-06-04 02:34:59,363.363 INFO:teuthology.lock.ops:unlocked smithi097.front.sepia.ceph.com 2021-06-04T02:36:59.666 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-06-04T02:36:59.784 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-06-04T02:36:59.785 INFO:teuthology.nuke:Checking targets against current locks 2021-06-04T02:36:59.873 INFO:teuthology.nuke:Not nuking smithi046.front.sepia.ceph.com because description doesn't match 2021-06-04T02:36:59.874 INFO:teuthology.nuke:Not nuking smithi097.front.sepia.ceph.com because description doesn't match