2021-06-02T18:54:38.596 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-02T18:54:38.695 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi029.front.sepia.ceph.com' reimaging 2021-06-02T18:54:38.696 INFO:teuthology.lock.ops:Updating [ubuntu@smithi029.front.sepia.ceph.com]: reset os type and version on server 2021-06-02T18:54:38.697 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2021-06-02T18:54:38.715 INFO:teuthology.lock.ops:Node 'ubuntu@smithi029.front.sepia.ceph.com' reimaging is complete 2021-06-02T18:54:38.716 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi164.front.sepia.ceph.com' reimaging 2021-06-02T18:54:38.716 INFO:teuthology.lock.ops:Updating [ubuntu@smithi164.front.sepia.ceph.com]: reset os type and version on server 2021-06-02T18:54:38.717 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2021-06-02T18:54:38.731 INFO:teuthology.lock.ops:Node 'ubuntu@smithi164.front.sepia.ceph.com' reimaging is complete 2021-06-02T18:54:38.930 INFO:teuthology.provision.fog.smithi029:Scheduling deploy of ubuntu 20.04 2021-06-02T18:54:38.941 INFO:teuthology.provision.fog.smithi164:Scheduling deploy of ubuntu 20.04 2021-06-02T18:54:39.418 INFO:teuthology.orchestra.console:Power off smithi164 2021-06-02T18:54:39.560 INFO:teuthology.orchestra.console:Power off smithi029 2021-06-02T18:54:47.788 INFO:teuthology.orchestra.console:Power off for smithi164 completed 2021-06-02T18:54:47.890 INFO:teuthology.orchestra.console:Power on smithi164 2021-06-02T18:54:47.932 INFO:teuthology.orchestra.console:Power off for smithi029 completed 2021-06-02T18:54:48.033 INFO:teuthology.orchestra.console:Power on smithi029 2021-06-02T18:54:56.258 INFO:teuthology.orchestra.console:Power on for smithi164 completed 2021-06-02T18:54:56.360 INFO:teuthology.provision.fog.smithi164:Waiting for deploy to finish 2021-06-02T18:54:56.397 INFO:teuthology.orchestra.console:Power on for smithi029 completed 2021-06-02T18:54:56.498 INFO:teuthology.provision.fog.smithi029:Waiting for deploy to finish 2021-06-02T19:00:07.521 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-02T19:00:07.945 INFO:teuthology.provision.fog.smithi164:Node is ready 2021-06-02T19:00:07.992 INFO:teuthology.orchestra.run.smithi164.stdout:smithi164.front.sepia.ceph.com 2021-06-02T19:00:07.998 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.15.164 smithi164.front.sepia.ceph.com smithi164 2021-06-02T19:00:08.195 INFO:teuthology.provision.fog.smithi164:Deploy complete! 2021-06-02T19:00:29.367 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-02T19:00:29.889 INFO:teuthology.provision.fog.smithi029:Node is ready 2021-06-02T19:00:29.939 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029.front.sepia.ceph.com 2021-06-02T19:00:29.947 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29 smithi029.front.sepia.ceph.com smithi029 2021-06-02T19:00:30.168 INFO:teuthology.provision.fog.smithi029:Deploy complete! 2021-06-02T19:00:30.326 INFO:teuthology.lock.ops:Checking smithi029.front.sepia.ceph.com 2021-06-02T19:00:30.329 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-02T19:00:30.350 INFO:teuthology.lock.ops:Checking smithi164.front.sepia.ceph.com 2021-06-02T19:00:30.350 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-02T19:00:30.367 INFO:teuthology.lock.ops:Updating [smithi029.front.sepia.ceph.com]: set os type and version on server 2021-06-02T19:00:30.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-02T19:00:30.547 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2021-06-02T19:00:30.552 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="Ubuntu" 2021-06-02T19:00:30.553 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-06-02T19:00:30.553 INFO:teuthology.orchestra.run.smithi029.stdout:ID=ubuntu 2021-06-02T19:00:30.554 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE=debian 2021-06-02T19:00:30.554 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-06-02T19:00:30.554 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="20.04" 2021-06-02T19:00:30.555 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://www.ubuntu.com/" 2021-06-02T19:00:30.555 INFO:teuthology.orchestra.run.smithi029.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-06-02T19:00:30.555 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-06-02T19:00:30.556 INFO:teuthology.orchestra.run.smithi029.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-06-02T19:00:30.556 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_CODENAME=focal 2021-06-02T19:00:30.556 INFO:teuthology.orchestra.run.smithi029.stdout:UBUNTU_CODENAME=focal 2021-06-02T19:00:30.557 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2021-06-02T19:00:30.573 INFO:teuthology.lock.ops:Updating [smithi164.front.sepia.ceph.com]: set os type and version on server 2021-06-02T19:00:30.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-02T19:00:30.787 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2021-06-02T19:00:30.795 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="Ubuntu" 2021-06-02T19:00:30.795 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-06-02T19:00:30.795 INFO:teuthology.orchestra.run.smithi164.stdout:ID=ubuntu 2021-06-02T19:00:30.796 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE=debian 2021-06-02T19:00:30.796 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-06-02T19:00:30.796 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="20.04" 2021-06-02T19:00:30.796 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://www.ubuntu.com/" 2021-06-02T19:00:30.797 INFO:teuthology.orchestra.run.smithi164.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-06-02T19:00:30.797 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-06-02T19:00:30.797 INFO:teuthology.orchestra.run.smithi164.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-06-02T19:00:30.797 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_CODENAME=focal 2021-06-02T19:00:30.798 INFO:teuthology.orchestra.run.smithi164.stdout:UBUNTU_CODENAME=focal 2021-06-02T19:00:30.798 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2021-06-02T19:00:30.890 INFO:teuthology.dispatcher.supervisor:Running job 6146246 2021-06-02T19:00:30.890 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2021-06-02_18:04:29-rados-wip-sage-testing-2021-06-02-1138-distro-basic-smithi/6146246 --name sage-2021-06-02_18:04:29-rados-wip-sage-testing-2021-06-02-1138-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/dedup-io-snaps} -- /home/teuthworker/archive/sage-2021-06-02_18:04:29-rados-wip-sage-testing-2021-06-02-1138-distro-basic-smithi/6146246/orig.config.yaml 2021-06-02T19:00:30.893 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sage-2021-06-02_18:04:29-rados-wip-sage-testing-2021-06-02-1138-distro-basic-smithi/6146246 2021-06-02T19:00:30.894 INFO:teuthology.dispatcher.supervisor:Job PID: 6939 2021-06-02T19:00:30.894 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-06-02T22:38:46.177 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/sage-2021-06-02_18:04:29-rados-wip-sage-testing-2021-06-02-1138-distro-basic-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2021-06-03T07:01:23.368 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-06-03T07:01:23.587 INFO:teuthology.kill:Killing Pids: {6939} 2021-06-03T07:01:24.796 INFO:teuthology.kill:Nuking machines: ['smithi029', 'smithi164'] 2021-06-03T07:01:25.863 INFO:teuthology.kill:2021-06-03 07:01:25,863.863 INFO:teuthology.nuke:targets: 2021-06-03T07:01:25.864 INFO:teuthology.kill: smithi029.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYQPjHmaBfQ84K5JiePlC43jAOqb/kP9UFp26do4bUMJmFje/KE86E+9yx/qx1TCGERIwWvwyyiPW6UovB+xAtBPVHONjChwOJj/leBySVBEa0NqnuXawQAoq7BQtF4VfYy3XEYYNiZcj4ce5RGYFRWfwq44d5argyPcjHShMjIEBkC8MzEQJXIFoXkkf0zuqX0KSHYDLzNH1J75Qt0QjCIWhoy7SCH0l72SvG4iL0VngGT+MpHT34Rczj2F9AfiCFCcymuW5EhexNt0yu09++ELuXra7MWuPda7Mnka0KuIDw4kG2Z8Mb9jncQhKUlcgGHFr4T/0Wbct3bxoIeEIgq0OPisc9su1yNV4W3H6xkDzuYGGf4bnrLuhRJE/Bkeig7UONnMmsDao53Duv8tXHPH15hHjSnVsR3J+4mzIhfgDTzgu18bpuFVHDrikOW8lSXhEHpn1qbEco0QfeHDQphLE3dJShV+sgZbF2NKCUWQbEreUqlAK8VQTZsrgwdXU= 2021-06-03T07:01:25.864 INFO:teuthology.kill: smithi164.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBdeBS5LeKUVXjEeQcXIZ6lHZMVssIHH4gBylg7GzzqQOUjU4LltI0v4bpq4GlGooYjI0ryg7ghbx4MM0XO0M7CzSm4KG+DElW5NOWGP7/UEVBXzgulDIWBxB98xQmLoyyqAEYfwyMCFJPWK1IUkWirUTTrjTdzA0KpsyfxyLRIJl1DRQj0YQP3U1Hy4en2ytuz5DeHRlbFw8avAgES803W4deP1p2apyuWL5mCeJV04yYrtmuOyb3Jz6JnIeAMcMSthTfDgyfh6aa6CkBV4b2ygYg5JdzCZw99e2vk55U2fGUyCFH1MF9KmfURbMOGSMO8vGDBcBD/XgqnLHEpE7SChV5uyCnuorD8EOad0XyLTe1Nw7cDjca7AjFHbnC4vK4tEkFwaf1FMgY5ZaSzb3qra4UW8EJiZK9I+uF+r7QA5Z62ot8VaWV7Xa//58vjCi6lByH8olPL34xAjAXSa1x2IlgaVGx/QXga0RI9xlqhC1m43pnSEo3cWrzVcbqUJk= 2021-06-03T07:01:25.865 INFO:teuthology.kill:2021-06-03 07:01:25,864.864 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-03T07:01:25.867 INFO:teuthology.kill:2021-06-03 07:01:25,867.867 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-03T07:01:25.937 INFO:teuthology.kill:2021-06-03 07:01:25,936.936 INFO:teuthology.orchestra.console:Power off smithi029 2021-06-03T07:01:25.952 INFO:teuthology.kill:2021-06-03 07:01:25,951.951 INFO:teuthology.orchestra.console:Power off smithi164 2021-06-03T07:01:34.309 INFO:teuthology.kill:2021-06-03 07:01:34,309.309 INFO:teuthology.orchestra.console:Power off for smithi029 completed 2021-06-03T07:01:38.430 INFO:teuthology.kill:2021-06-03 07:01:38,430.430 INFO:teuthology.orchestra.console:Power off for smithi164 completed 2021-06-03T07:01:38.676 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-06-03T07:01:38.677 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-06-03T07:01:38.677 INFO:teuthology.misc:Compressing logs... 2021-06-03T07:01:38.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-03T07:02:13.199 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-06-03T07:02:13.260 INFO:teuthology.kill:No teuthology processes running 2021-06-03T07:02:14.351 INFO:teuthology.kill:Nuking machines: ['smithi029', 'smithi164'] 2021-06-03T07:02:15.322 INFO:teuthology.kill:2021-06-03 07:02:15,322.322 INFO:teuthology.nuke:targets: 2021-06-03T07:02:15.323 INFO:teuthology.kill: smithi029.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYQPjHmaBfQ84K5JiePlC43jAOqb/kP9UFp26do4bUMJmFje/KE86E+9yx/qx1TCGERIwWvwyyiPW6UovB+xAtBPVHONjChwOJj/leBySVBEa0NqnuXawQAoq7BQtF4VfYy3XEYYNiZcj4ce5RGYFRWfwq44d5argyPcjHShMjIEBkC8MzEQJXIFoXkkf0zuqX0KSHYDLzNH1J75Qt0QjCIWhoy7SCH0l72SvG4iL0VngGT+MpHT34Rczj2F9AfiCFCcymuW5EhexNt0yu09++ELuXra7MWuPda7Mnka0KuIDw4kG2Z8Mb9jncQhKUlcgGHFr4T/0Wbct3bxoIeEIgq0OPisc9su1yNV4W3H6xkDzuYGGf4bnrLuhRJE/Bkeig7UONnMmsDao53Duv8tXHPH15hHjSnVsR3J+4mzIhfgDTzgu18bpuFVHDrikOW8lSXhEHpn1qbEco0QfeHDQphLE3dJShV+sgZbF2NKCUWQbEreUqlAK8VQTZsrgwdXU= 2021-06-03T07:02:15.324 INFO:teuthology.kill: smithi164.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBdeBS5LeKUVXjEeQcXIZ6lHZMVssIHH4gBylg7GzzqQOUjU4LltI0v4bpq4GlGooYjI0ryg7ghbx4MM0XO0M7CzSm4KG+DElW5NOWGP7/UEVBXzgulDIWBxB98xQmLoyyqAEYfwyMCFJPWK1IUkWirUTTrjTdzA0KpsyfxyLRIJl1DRQj0YQP3U1Hy4en2ytuz5DeHRlbFw8avAgES803W4deP1p2apyuWL5mCeJV04yYrtmuOyb3Jz6JnIeAMcMSthTfDgyfh6aa6CkBV4b2ygYg5JdzCZw99e2vk55U2fGUyCFH1MF9KmfURbMOGSMO8vGDBcBD/XgqnLHEpE7SChV5uyCnuorD8EOad0XyLTe1Nw7cDjca7AjFHbnC4vK4tEkFwaf1FMgY5ZaSzb3qra4UW8EJiZK9I+uF+r7QA5Z62ot8VaWV7Xa//58vjCi6lByH8olPL34xAjAXSa1x2IlgaVGx/QXga0RI9xlqhC1m43pnSEo3cWrzVcbqUJk= 2021-06-03T07:02:15.346 INFO:teuthology.kill:2021-06-03 07:02:15,345.345 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-03T07:02:15.354 INFO:teuthology.kill:2021-06-03 07:02:15,353.353 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-03T07:02:15.440 INFO:teuthology.kill:2021-06-03 07:02:15,440.440 INFO:teuthology.orchestra.console:Power off smithi164 2021-06-03T07:02:15.461 INFO:teuthology.kill:2021-06-03 07:02:15,461.461 INFO:teuthology.orchestra.console:Power off smithi029 2021-06-03T07:02:15.682 INFO:teuthology.kill:2021-06-03 07:02:15,681.681 INFO:teuthology.orchestra.console:Power off for smithi029 completed 2021-06-03T07:02:15.713 INFO:teuthology.kill:2021-06-03 07:02:15,713.713 INFO:teuthology.orchestra.console:Power off for smithi164 completed 2021-06-03T07:02:15.831 INFO:teuthology.kill:2021-06-03 07:02:15,828.828 INFO:teuthology.lock.ops:unlocked smithi029.front.sepia.ceph.com 2021-06-03T07:02:15.860 INFO:teuthology.kill:2021-06-03 07:02:15,860.860 INFO:teuthology.lock.ops:unlocked smithi164.front.sepia.ceph.com 2021-06-03T07:04:16.186 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-06-03T07:04:16.330 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-06-03T07:04:16.330 INFO:teuthology.nuke:Checking targets against current locks 2021-06-03T07:04:16.401 INFO:teuthology.nuke:Not nuking smithi164.front.sepia.ceph.com because description doesn't match 2021-06-03T07:04:16.402 INFO:teuthology.nuke:Not nuking smithi029.front.sepia.ceph.com because description doesn't match