2021-05-01T05:57:44.684 INFO:root:teuthology version: 1.1.0-bea5b739 2021-05-01T05:57:44.785 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi114.front.sepia.ceph.com' reimaging 2021-05-01T05:57:44.785 INFO:teuthology.lock.ops:Updating [ubuntu@smithi114.front.sepia.ceph.com]: reset os type and version on server 2021-05-01T05:57:44.785 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2021-05-01T05:57:44.801 INFO:teuthology.lock.ops:Node 'ubuntu@smithi114.front.sepia.ceph.com' reimaging is complete 2021-05-01T05:57:44.802 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi185.front.sepia.ceph.com' reimaging 2021-05-01T05:57:44.802 INFO:teuthology.lock.ops:Updating [ubuntu@smithi185.front.sepia.ceph.com]: reset os type and version on server 2021-05-01T05:57:44.802 INFO:teuthology.lock.ops:Updating smithi185.front.sepia.ceph.com on lock server 2021-05-01T05:57:44.818 INFO:teuthology.lock.ops:Node 'ubuntu@smithi185.front.sepia.ceph.com' reimaging is complete 2021-05-01T05:57:44.998 INFO:teuthology.provision.fog.smithi114:Scheduling deploy of ubuntu 20.04 2021-05-01T05:57:45.019 INFO:teuthology.provision.fog.smithi185:Scheduling deploy of ubuntu 20.04 2021-05-01T05:57:45.339 INFO:teuthology.orchestra.console:Power off smithi185 2021-05-01T05:57:45.357 INFO:teuthology.orchestra.console:Power off smithi114 2021-05-01T05:57:53.713 INFO:teuthology.orchestra.console:Power off for smithi185 completed 2021-05-01T05:57:53.814 INFO:teuthology.orchestra.console:Power on smithi185 2021-05-01T05:57:57.834 INFO:teuthology.orchestra.console:Power off for smithi114 completed 2021-05-01T05:57:57.935 INFO:teuthology.orchestra.console:Power on smithi114 2021-05-01T05:58:02.188 INFO:teuthology.orchestra.console:Power on for smithi114 completed 2021-05-01T05:58:02.246 INFO:teuthology.orchestra.console:Power on for smithi185 completed 2021-05-01T05:58:02.289 INFO:teuthology.provision.fog.smithi114:Waiting for deploy to finish 2021-05-01T05:58:02.347 INFO:teuthology.provision.fog.smithi185:Waiting for deploy to finish 2021-05-01T06:02:59.881 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-01T06:03:00.280 INFO:teuthology.provision.fog.smithi114:Node is ready 2021-05-01T06:03:00.287 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114.front.sepia.ceph.com 2021-05-01T06:03:00.294 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114 smithi114.front.sepia.ceph.com smithi114 2021-05-01T06:03:00.549 INFO:teuthology.provision.fog.smithi114:Deploy complete! 2021-05-01T06:03:18.090 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-01T06:03:18.591 INFO:teuthology.provision.fog.smithi185:Node is ready 2021-05-01T06:03:18.610 INFO:teuthology.orchestra.run.smithi185.stdout:smithi185.front.sepia.ceph.com 2021-05-01T06:03:18.668 INFO:teuthology.orchestra.run.smithi185.stdout:172.21.15.185 smithi185.front.sepia.ceph.com smithi185 2021-05-01T06:03:18.987 INFO:teuthology.provision.fog.smithi185:Deploy complete! 2021-05-01T06:03:19.184 INFO:teuthology.lock.ops:Checking smithi114.front.sepia.ceph.com 2021-05-01T06:03:19.185 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-01T06:03:19.209 INFO:teuthology.lock.ops:Checking smithi185.front.sepia.ceph.com 2021-05-01T06:03:19.209 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-01T06:03:19.231 INFO:teuthology.lock.ops:Updating [smithi114.front.sepia.ceph.com]: set os type and version on server 2021-05-01T06:03:19.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-01T06:03:19.426 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2021-05-01T06:03:19.476 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="Ubuntu" 2021-05-01T06:03:19.476 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-01T06:03:19.476 INFO:teuthology.orchestra.run.smithi114.stdout:ID=ubuntu 2021-05-01T06:03:19.476 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE=debian 2021-05-01T06:03:19.476 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-01T06:03:19.476 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="20.04" 2021-05-01T06:03:19.476 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-01T06:03:19.476 INFO:teuthology.orchestra.run.smithi114.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-01T06:03:19.477 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-01T06:03:19.477 INFO:teuthology.orchestra.run.smithi114.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-01T06:03:19.477 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_CODENAME=focal 2021-05-01T06:03:19.477 INFO:teuthology.orchestra.run.smithi114.stdout:UBUNTU_CODENAME=focal 2021-05-01T06:03:19.478 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2021-05-01T06:03:19.497 INFO:teuthology.lock.ops:Updating [smithi185.front.sepia.ceph.com]: set os type and version on server 2021-05-01T06:03:19.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-01T06:03:19.719 INFO:teuthology.orchestra.run.smithi185.stdout:x86_64 2021-05-01T06:03:19.725 INFO:teuthology.orchestra.run.smithi185.stdout:NAME="Ubuntu" 2021-05-01T06:03:19.725 INFO:teuthology.orchestra.run.smithi185.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-01T06:03:19.725 INFO:teuthology.orchestra.run.smithi185.stdout:ID=ubuntu 2021-05-01T06:03:19.725 INFO:teuthology.orchestra.run.smithi185.stdout:ID_LIKE=debian 2021-05-01T06:03:19.725 INFO:teuthology.orchestra.run.smithi185.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-01T06:03:19.725 INFO:teuthology.orchestra.run.smithi185.stdout:VERSION_ID="20.04" 2021-05-01T06:03:19.725 INFO:teuthology.orchestra.run.smithi185.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-01T06:03:19.725 INFO:teuthology.orchestra.run.smithi185.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-01T06:03:19.726 INFO:teuthology.orchestra.run.smithi185.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-01T06:03:19.726 INFO:teuthology.orchestra.run.smithi185.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-01T06:03:19.726 INFO:teuthology.orchestra.run.smithi185.stdout:VERSION_CODENAME=focal 2021-05-01T06:03:19.726 INFO:teuthology.orchestra.run.smithi185.stdout:UBUNTU_CODENAME=focal 2021-05-01T06:03:19.726 INFO:teuthology.lock.ops:Updating smithi185.front.sepia.ceph.com on lock server 2021-05-01T06:03:19.813 INFO:teuthology.dispatcher.supervisor:Running job 6082219 2021-05-01T06:03:19.813 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-04-29_13:53:41-rados-benhanokh-NCB-distro-basic-smithi/6082219 --name benhanokh-2021-04-29_13:53:41-rados-benhanokh-NCB-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect} -- /home/teuthworker/archive/benhanokh-2021-04-29_13:53:41-rados-benhanokh-NCB-distro-basic-smithi/6082219/orig.config.yaml 2021-05-01T06:03:19.817 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-04-29_13:53:41-rados-benhanokh-NCB-distro-basic-smithi/6082219 2021-05-01T06:03:19.817 INFO:teuthology.dispatcher.supervisor:Job PID: 19723 2021-05-01T06:03:19.817 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-01T07:31:25.722 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/benhanokh-2021-04-29_13:53:41-rados-benhanokh-NCB-distro-basic-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2021-05-01T18:04:10.256 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-01T18:04:10.463 INFO:teuthology.kill:Killing Pids: {19723} 2021-05-01T18:04:11.539 INFO:teuthology.kill:Nuking machines: ['smithi114', 'smithi185'] 2021-05-01T18:04:12.483 INFO:teuthology.kill:2021-05-01 18:04:12,483.483 INFO:teuthology.nuke:targets: 2021-05-01T18:04:12.484 INFO:teuthology.kill: smithi114.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIYZ6hSuUZUaXvJ+Pl8rsO1jlsAMCvuyYQwUTcDCR7E+gxfKIptCMFl/x0c1sbQihpvzW4JpZIKdRi61A39DsR+gL95BFot00nz+Fxeh2vgj1YU8IUea3BjE9/FMMymaJYp4SbAGUX4iyrK5BE2lfph/lCxxNDYiGnKwW2Hr/BkAQJLwdb4xcBkDz5xpAPSBADJcoCju9MbPJiVDijAoTIMJfMTN9OJHB2QIxzgCMvZ57GIXY9eg2Pdxxl1rRXgV7PAGGmLxtMI6VQlTAQR7Vi9ES7b5qqlhT5m09t3PahF8zxqEpB0IiRZDEDenRyntKRYJRhu87mbhrqA+qFg+FDf3MyFtSUPi4ZsanuPgByBFAi0zpiQlddKTV2GWpyiup/WzmgvdjQH3g9rIHSgQoz1Ty2PUcJD/sGYNXIS0SO5hlPzKgEtkA2dS6JDNecccc3vnL975y+hAe9uwirYfIYZgmB0vfaAe3QxzvbPkX63SXYFUdtmXOTEyPMDVPsm3k= 2021-05-01T18:04:12.484 INFO:teuthology.kill: smithi185.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMXeC1hPqW2IKGAoAEEVvPmrmAbkCZKx6SBF3rxtGcuHo2Zxph8PkWRnLbfbuxufqXWJGmNqIftQ16JpxOU436TIU0HlxelUwg/JnLXkgFdY9Hy9OK3sWmuWZKagnZkS+rLvxunmi5tbSlBLSGSw+mGEYJdO4IVkh4o7atTYG2oDZYl0qwOj5MEp3VIiq3VVNfQrzC0g+JW56URqHACB9xoH0j+Ahz2iF/FmxhdZufOi4yVgjLCqsQXyM0KpGQsCzx+M7lLA9Rg3y21sx8f/RtBCt+HBYJm1GlkuCKQPkt7ZYWCjsK1blzvA6BnIk53uOoww18lp5ii5cSLIP9eqkvVQ52QrdMOKjYtV9PZfgWOpr9c3VF5Y0t1Vc3pGx0tM550ABk8iK/+sWiHTNS67X2PHnejiIqhI+gD29mTmVIj1ZW81v34bKbNkY4eq0Ikc7ExeEJEaDkByg7AoV3pox3+YSiKN9V/aHPprNURGPbSeamqUuhz5bZySoBsABBHks= 2021-05-01T18:04:12.484 INFO:teuthology.kill:2021-05-01 18:04:12,483.483 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-01T18:04:12.486 INFO:teuthology.kill:2021-05-01 18:04:12,486.486 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-01T18:04:12.550 INFO:teuthology.kill:2021-05-01 18:04:12,550.550 INFO:teuthology.orchestra.console:Power off smithi185 2021-05-01T18:04:12.563 INFO:teuthology.kill:2021-05-01 18:04:12,563.563 INFO:teuthology.orchestra.console:Power off smithi114 2021-05-01T18:04:20.918 INFO:teuthology.kill:2021-05-01 18:04:20,918.918 INFO:teuthology.orchestra.console:Power off for smithi185 completed 2021-05-01T18:04:25.037 INFO:teuthology.kill:2021-05-01 18:04:25,036.036 INFO:teuthology.orchestra.console:Power off for smithi114 completed 2021-05-01T18:04:25.303 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-05-01T18:04:25.303 INFO:teuthology.task.internal:roles: ubuntu@smithi185.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-05-01T18:04:25.304 INFO:teuthology.misc:Compressing logs... 2021-05-01T18:04:25.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-01T18:05:25.366 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-01T18:05:25.424 INFO:teuthology.kill:No teuthology processes running 2021-05-01T18:05:26.530 INFO:teuthology.kill:Nuking machines: ['smithi114', 'smithi185'] 2021-05-01T18:05:27.462 INFO:teuthology.kill:2021-05-01 18:05:27,462.462 INFO:teuthology.nuke:targets: 2021-05-01T18:05:27.463 INFO:teuthology.kill: smithi114.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIYZ6hSuUZUaXvJ+Pl8rsO1jlsAMCvuyYQwUTcDCR7E+gxfKIptCMFl/x0c1sbQihpvzW4JpZIKdRi61A39DsR+gL95BFot00nz+Fxeh2vgj1YU8IUea3BjE9/FMMymaJYp4SbAGUX4iyrK5BE2lfph/lCxxNDYiGnKwW2Hr/BkAQJLwdb4xcBkDz5xpAPSBADJcoCju9MbPJiVDijAoTIMJfMTN9OJHB2QIxzgCMvZ57GIXY9eg2Pdxxl1rRXgV7PAGGmLxtMI6VQlTAQR7Vi9ES7b5qqlhT5m09t3PahF8zxqEpB0IiRZDEDenRyntKRYJRhu87mbhrqA+qFg+FDf3MyFtSUPi4ZsanuPgByBFAi0zpiQlddKTV2GWpyiup/WzmgvdjQH3g9rIHSgQoz1Ty2PUcJD/sGYNXIS0SO5hlPzKgEtkA2dS6JDNecccc3vnL975y+hAe9uwirYfIYZgmB0vfaAe3QxzvbPkX63SXYFUdtmXOTEyPMDVPsm3k= 2021-05-01T18:05:27.463 INFO:teuthology.kill: smithi185.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMXeC1hPqW2IKGAoAEEVvPmrmAbkCZKx6SBF3rxtGcuHo2Zxph8PkWRnLbfbuxufqXWJGmNqIftQ16JpxOU436TIU0HlxelUwg/JnLXkgFdY9Hy9OK3sWmuWZKagnZkS+rLvxunmi5tbSlBLSGSw+mGEYJdO4IVkh4o7atTYG2oDZYl0qwOj5MEp3VIiq3VVNfQrzC0g+JW56URqHACB9xoH0j+Ahz2iF/FmxhdZufOi4yVgjLCqsQXyM0KpGQsCzx+M7lLA9Rg3y21sx8f/RtBCt+HBYJm1GlkuCKQPkt7ZYWCjsK1blzvA6BnIk53uOoww18lp5ii5cSLIP9eqkvVQ52QrdMOKjYtV9PZfgWOpr9c3VF5Y0t1Vc3pGx0tM550ABk8iK/+sWiHTNS67X2PHnejiIqhI+gD29mTmVIj1ZW81v34bKbNkY4eq0Ikc7ExeEJEaDkByg7AoV3pox3+YSiKN9V/aHPprNURGPbSeamqUuhz5bZySoBsABBHks= 2021-05-01T18:05:27.484 INFO:teuthology.kill:2021-05-01 18:05:27,484.484 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-01T18:05:27.488 INFO:teuthology.kill:2021-05-01 18:05:27,488.488 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-01T18:05:27.549 INFO:teuthology.kill:2021-05-01 18:05:27,548.548 INFO:teuthology.orchestra.console:Power off smithi185 2021-05-01T18:05:27.564 INFO:teuthology.kill:2021-05-01 18:05:27,563.563 INFO:teuthology.orchestra.console:Power off smithi114 2021-05-01T18:05:27.693 INFO:teuthology.kill:2021-05-01 18:05:27,693.693 INFO:teuthology.orchestra.console:Power off for smithi185 completed 2021-05-01T18:05:27.706 INFO:teuthology.kill:2021-05-01 18:05:27,706.706 INFO:teuthology.orchestra.console:Power off for smithi114 completed 2021-05-01T18:05:27.839 INFO:teuthology.kill:2021-05-01 18:05:27,838.838 INFO:teuthology.lock.ops:unlocked smithi185.front.sepia.ceph.com 2021-05-01T18:05:27.851 INFO:teuthology.kill:2021-05-01 18:05:27,851.851 INFO:teuthology.lock.ops:unlocked smithi114.front.sepia.ceph.com 2021-05-01T18:07:28.136 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-05-01T18:07:28.226 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-05-01T18:07:28.227 INFO:teuthology.nuke:Checking targets against current locks 2021-05-01T18:07:28.301 INFO:teuthology.nuke:Not nuking smithi114.front.sepia.ceph.com because description doesn't match 2021-05-01T18:07:28.301 INFO:teuthology.nuke:Not nuking smithi185.front.sepia.ceph.com because description doesn't match