2021-06-30T08:30:09.103 INFO:root:teuthology version: 1.1.0-26631ede 2021-06-30T08:30:09.237 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi138.front.sepia.ceph.com' reimaging 2021-06-30T08:30:09.238 INFO:teuthology.lock.ops:Updating [ubuntu@smithi138.front.sepia.ceph.com]: reset os type and version on server 2021-06-30T08:30:09.238 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2021-06-30T08:30:09.255 INFO:teuthology.lock.ops:Node 'ubuntu@smithi138.front.sepia.ceph.com' reimaging is complete 2021-06-30T08:30:09.255 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi149.front.sepia.ceph.com' reimaging 2021-06-30T08:30:09.255 INFO:teuthology.lock.ops:Updating [ubuntu@smithi149.front.sepia.ceph.com]: reset os type and version on server 2021-06-30T08:30:09.256 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2021-06-30T08:30:09.274 INFO:teuthology.lock.ops:Node 'ubuntu@smithi149.front.sepia.ceph.com' reimaging is complete 2021-06-30T08:30:09.506 INFO:teuthology.provision.fog.smithi138:Scheduling deploy of centos 8.3 2021-06-30T08:30:09.511 INFO:teuthology.provision.fog.smithi149:Scheduling deploy of centos 8.3 2021-06-30T08:30:09.868 INFO:teuthology.orchestra.console:Power off smithi149 2021-06-30T08:30:09.887 INFO:teuthology.orchestra.console:Power off smithi138 2021-06-30T08:30:18.265 INFO:teuthology.orchestra.console:Power off for smithi149 completed 2021-06-30T08:30:18.286 INFO:teuthology.orchestra.console:Power off for smithi138 completed 2021-06-30T08:30:18.367 INFO:teuthology.orchestra.console:Power on smithi149 2021-06-30T08:30:18.388 INFO:teuthology.orchestra.console:Power on smithi138 2021-06-30T08:30:26.774 INFO:teuthology.orchestra.console:Power on for smithi138 completed 2021-06-30T08:30:26.810 INFO:teuthology.orchestra.console:Power on for smithi149 completed 2021-06-30T08:30:26.876 INFO:teuthology.provision.fog.smithi138:Waiting for deploy to finish 2021-06-30T08:30:26.911 INFO:teuthology.provision.fog.smithi149:Waiting for deploy to finish 2021-06-30T08:34:58.973 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-30T08:34:59.520 INFO:teuthology.provision.fog.smithi138:Node is ready 2021-06-30T08:34:59.536 INFO:teuthology.orchestra.run.smithi138.stdout:smithi138.front.sepia.ceph.com 2021-06-30T08:34:59.592 INFO:teuthology.orchestra.run.smithi138.stdout:172.21.15.138 smithi138.front.sepia.ceph.com smithi138 2021-06-30T08:34:59.922 INFO:teuthology.provision.fog.smithi138:Deploy complete! 2021-06-30T08:35:07.995 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-30T08:35:08.360 INFO:teuthology.provision.fog.smithi149:Node is ready 2021-06-30T08:35:08.381 INFO:teuthology.orchestra.run.smithi149.stdout:smithi149.front.sepia.ceph.com 2021-06-30T08:35:08.436 INFO:teuthology.orchestra.run.smithi149.stdout:172.21.15.149 smithi149.front.sepia.ceph.com smithi149 2021-06-30T08:35:08.753 INFO:teuthology.provision.fog.smithi149:Deploy complete! 2021-06-30T08:35:08.889 INFO:teuthology.lock.ops:Checking smithi138.front.sepia.ceph.com 2021-06-30T08:35:08.889 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-30T08:35:08.911 INFO:teuthology.lock.ops:Checking smithi149.front.sepia.ceph.com 2021-06-30T08:35:08.911 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-30T08:35:08.932 INFO:teuthology.lock.ops:Updating [smithi138.front.sepia.ceph.com]: set os type and version on server 2021-06-30T08:35:08.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-30T08:35:09.230 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2021-06-30T08:35:09.286 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="CentOS Linux" 2021-06-30T08:35:09.286 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="8" 2021-06-30T08:35:09.286 INFO:teuthology.orchestra.run.smithi138.stdout:ID="centos" 2021-06-30T08:35:09.286 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="rhel fedora" 2021-06-30T08:35:09.287 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="8" 2021-06-30T08:35:09.287 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el8" 2021-06-30T08:35:09.287 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="CentOS Linux 8" 2021-06-30T08:35:09.287 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31" 2021-06-30T08:35:09.288 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-30T08:35:09.288 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://centos.org/" 2021-06-30T08:35:09.288 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-30T08:35:09.288 INFO:teuthology.orchestra.run.smithi138.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-30T08:35:09.289 INFO:teuthology.orchestra.run.smithi138.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-30T08:35:09.292 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2021-06-30T08:35:09.312 INFO:teuthology.lock.ops:Updating [smithi149.front.sepia.ceph.com]: set os type and version on server 2021-06-30T08:35:09.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-30T08:35:09.638 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64 2021-06-30T08:35:09.693 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="CentOS Linux" 2021-06-30T08:35:09.693 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="8" 2021-06-30T08:35:09.693 INFO:teuthology.orchestra.run.smithi149.stdout:ID="centos" 2021-06-30T08:35:09.694 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="rhel fedora" 2021-06-30T08:35:09.694 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="8" 2021-06-30T08:35:09.694 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el8" 2021-06-30T08:35:09.695 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="CentOS Linux 8" 2021-06-30T08:35:09.695 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31" 2021-06-30T08:35:09.695 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-30T08:35:09.695 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://centos.org/" 2021-06-30T08:35:09.695 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-30T08:35:09.696 INFO:teuthology.orchestra.run.smithi149.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-30T08:35:09.696 INFO:teuthology.orchestra.run.smithi149.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-30T08:35:09.697 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2021-06-30T08:35:09.812 INFO:teuthology.dispatcher.supervisor:Running job 6209240 2021-06-30T08:35:09.813 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-06-26_19:32:15-rados-wip-ronenf-scrubs-config-distro-basic-smithi/6209240 --name rfriedma-2021-06-26_19:32:15-rados-wip-ronenf-scrubs-config-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-agent-big} -- /home/teuthworker/archive/rfriedma-2021-06-26_19:32:15-rados-wip-ronenf-scrubs-config-distro-basic-smithi/6209240/orig.config.yaml 2021-06-30T08:35:09.818 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-06-26_19:32:15-rados-wip-ronenf-scrubs-config-distro-basic-smithi/6209240 2021-06-30T08:35:09.819 INFO:teuthology.dispatcher.supervisor:Job PID: 26713 2021-06-30T08:35:09.819 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-06-30T12:39:28.062 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/rfriedma-2021-06-26_19:32:15-rados-wip-ronenf-scrubs-config-distro-basic-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2021-06-30T20:36:08.059 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-06-30T20:36:08.277 INFO:teuthology.kill:Killing Pids: {26713} 2021-06-30T20:36:09.504 INFO:teuthology.kill:Nuking machines: ['smithi138', 'smithi149'] 2021-06-30T20:36:10.557 INFO:teuthology.kill:2021-06-30 20:36:10,557.557 INFO:teuthology.nuke:targets: 2021-06-30T20:36:10.558 INFO:teuthology.kill: smithi138.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMkKgu/+TycKChHQgyC1niXKM24m8QNPIe7uT5NubZc/k734hkNl+rVRvCRkfQncOA0zfN8C98GmMCYeyF70MtqqvMs4VdKPGQ1nc++cjEFqW0WHEkbUHxzgGQuNn+5hAeejcQ3CgMoeEoPB4p7JjdHqmHqzmkerpzrB4cSd2C+IShYfOTSy3SigPHyPB/ywBLu5STMMMjlj+SXrGRJqenD8hBmOI1yYPdDfp0q9Rrs0zDk7C6kGrylmzBjI8DAnDigHWdIQjtAEwUrbnG6ZpHE8ZAVY8yK7TVLoGmu+oh0moFARg7eUU/K3OQm7fkfgypG3xXYpFvQ3a1feGDLU6f6/lYgQ2H1E/XBDcPC++nR7iCBaQ7BYTQmMTzprn/kZRb3Fz7rsG3wI57cdQUysyX9gB4qOmNXpBkAZKecJQyw1auOzX216RIT8B+bOfz2UQenI1Fc0OZd0vk/+6IZCnNNmilDQqSMnalzbxlcf/fMw63TU1i5T23R+W4TYOJVF0= 2021-06-30T20:36:10.558 INFO:teuthology.kill: smithi149.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+Q162Ss8JoIJiSAgxiBPLxOeanM1YmrNSUqaXYc1w9rIsah/biBybRVVL6g9WYc0bhgFcUGUr7MHmWmBoyDwiAjCMOPZ58pjWUzfVSaLCf56m3Q9W1WlM3NJheyxp5K8q+p8D5ZPs4Jw83Xo60LzZncLilc8fzicS9pyyZqIHqndOJLENjdNQ79IpCdKmRhCPKtRU4fLHfYni1aemqNN057Hz4h2hBhb+0Vdfivb7wQzoT1P/Co+4XUCI3TGRzClcw9Xts58y8ubhQ1iWFFLSdCfNJilQqlqLezCmv6b8tMu422v3cf2z/l/2bg0b3563UOMzqImhovBj3FUpSq9KYPDxSLueNDPOBm/Uw+yVrz3pjgRJAraevwZX0uuZvxlPc3HfBsy4pzA4i9FgyXTLzOmGma7tk0pOzIL6zbeSkUFcwByklXfLeA3LceodnwrvJXq9AX59Ldbw3w5jzebKkSgSso8IpVYut2rgDHCucermPflrzwQr0DpJxsMJgEs= 2021-06-30T20:36:10.559 INFO:teuthology.kill:2021-06-30 20:36:10,558.558 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-30T20:36:10.562 INFO:teuthology.kill:2021-06-30 20:36:10,562.562 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-30T20:36:10.633 INFO:teuthology.kill:2021-06-30 20:36:10,633.633 INFO:teuthology.orchestra.console:Power off smithi138 2021-06-30T20:36:10.648 INFO:teuthology.kill:2021-06-30 20:36:10,648.648 INFO:teuthology.orchestra.console:Power off smithi149 2021-06-30T20:36:19.012 INFO:teuthology.kill:2021-06-30 20:36:19,011.011 INFO:teuthology.orchestra.console:Power off for smithi138 completed 2021-06-30T20:36:19.021 INFO:teuthology.kill:2021-06-30 20:36:19,021.021 INFO:teuthology.orchestra.console:Power off for smithi149 completed 2021-06-30T20:36:19.288 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-06-30T20:36:19.289 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-06-30T20:36:19.289 INFO:teuthology.misc:Compressing logs... 2021-06-30T20:36:19.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-30T20:36:53.835 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-06-30T20:36:53.916 INFO:teuthology.kill:No teuthology processes running 2021-06-30T20:36:55.391 INFO:teuthology.kill:Nuking machines: ['smithi138', 'smithi149'] 2021-06-30T20:36:56.435 INFO:teuthology.kill:2021-06-30 20:36:56,434.434 INFO:teuthology.nuke:targets: 2021-06-30T20:36:56.436 INFO:teuthology.kill: smithi138.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMkKgu/+TycKChHQgyC1niXKM24m8QNPIe7uT5NubZc/k734hkNl+rVRvCRkfQncOA0zfN8C98GmMCYeyF70MtqqvMs4VdKPGQ1nc++cjEFqW0WHEkbUHxzgGQuNn+5hAeejcQ3CgMoeEoPB4p7JjdHqmHqzmkerpzrB4cSd2C+IShYfOTSy3SigPHyPB/ywBLu5STMMMjlj+SXrGRJqenD8hBmOI1yYPdDfp0q9Rrs0zDk7C6kGrylmzBjI8DAnDigHWdIQjtAEwUrbnG6ZpHE8ZAVY8yK7TVLoGmu+oh0moFARg7eUU/K3OQm7fkfgypG3xXYpFvQ3a1feGDLU6f6/lYgQ2H1E/XBDcPC++nR7iCBaQ7BYTQmMTzprn/kZRb3Fz7rsG3wI57cdQUysyX9gB4qOmNXpBkAZKecJQyw1auOzX216RIT8B+bOfz2UQenI1Fc0OZd0vk/+6IZCnNNmilDQqSMnalzbxlcf/fMw63TU1i5T23R+W4TYOJVF0= 2021-06-30T20:36:56.436 INFO:teuthology.kill: smithi149.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+Q162Ss8JoIJiSAgxiBPLxOeanM1YmrNSUqaXYc1w9rIsah/biBybRVVL6g9WYc0bhgFcUGUr7MHmWmBoyDwiAjCMOPZ58pjWUzfVSaLCf56m3Q9W1WlM3NJheyxp5K8q+p8D5ZPs4Jw83Xo60LzZncLilc8fzicS9pyyZqIHqndOJLENjdNQ79IpCdKmRhCPKtRU4fLHfYni1aemqNN057Hz4h2hBhb+0Vdfivb7wQzoT1P/Co+4XUCI3TGRzClcw9Xts58y8ubhQ1iWFFLSdCfNJilQqlqLezCmv6b8tMu422v3cf2z/l/2bg0b3563UOMzqImhovBj3FUpSq9KYPDxSLueNDPOBm/Uw+yVrz3pjgRJAraevwZX0uuZvxlPc3HfBsy4pzA4i9FgyXTLzOmGma7tk0pOzIL6zbeSkUFcwByklXfLeA3LceodnwrvJXq9AX59Ldbw3w5jzebKkSgSso8IpVYut2rgDHCucermPflrzwQr0DpJxsMJgEs= 2021-06-30T20:36:56.456 INFO:teuthology.kill:2021-06-30 20:36:56,456.456 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-30T20:36:56.460 INFO:teuthology.kill:2021-06-30 20:36:56,460.460 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-30T20:36:56.517 INFO:teuthology.kill:2021-06-30 20:36:56,517.517 INFO:teuthology.orchestra.console:Power off smithi149 2021-06-30T20:36:56.533 INFO:teuthology.kill:2021-06-30 20:36:56,533.533 INFO:teuthology.orchestra.console:Power off smithi138 2021-06-30T20:36:56.665 INFO:teuthology.kill:2021-06-30 20:36:56,665.665 INFO:teuthology.orchestra.console:Power off for smithi149 completed 2021-06-30T20:36:56.677 INFO:teuthology.kill:2021-06-30 20:36:56,676.676 INFO:teuthology.orchestra.console:Power off for smithi138 completed 2021-06-30T20:36:56.818 INFO:teuthology.kill:2021-06-30 20:36:56,817.817 INFO:teuthology.lock.ops:unlocked smithi149.front.sepia.ceph.com 2021-06-30T20:36:56.836 INFO:teuthology.kill:2021-06-30 20:36:56,836.836 INFO:teuthology.lock.ops:unlocked smithi138.front.sepia.ceph.com 2021-06-30T20:38:57.237 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-06-30T20:38:57.336 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-06-30T20:38:57.337 INFO:teuthology.nuke:Checking targets against current locks 2021-06-30T20:38:57.435 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-30T20:38:57.437 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-30T20:38:57.503 INFO:teuthology.orchestra.console:Power off smithi149 2021-06-30T20:38:57.519 INFO:teuthology.orchestra.console:Power off smithi138 2021-06-30T20:39:05.896 INFO:teuthology.orchestra.console:Power off for smithi138 completed 2021-06-30T20:39:06.046 INFO:teuthology.lock.ops:unlocked smithi138.front.sepia.ceph.com 2021-06-30T20:39:10.006 INFO:teuthology.orchestra.console:Power off for smithi149 completed 2021-06-30T20:39:10.147 INFO:teuthology.lock.ops:unlocked smithi149.front.sepia.ceph.com