2021-06-26T23:54:05.818 INFO:root:teuthology version: 1.1.0-26631ede 2021-06-26T23:54:06.052 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi022.front.sepia.ceph.com' reimaging 2021-06-26T23:54:06.053 INFO:teuthology.lock.ops:Updating [ubuntu@smithi022.front.sepia.ceph.com]: reset os type and version on server 2021-06-26T23:54:06.053 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2021-06-26T23:54:06.077 INFO:teuthology.lock.ops:Node 'ubuntu@smithi022.front.sepia.ceph.com' reimaging is complete 2021-06-26T23:54:06.077 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi038.front.sepia.ceph.com' reimaging 2021-06-26T23:54:06.078 INFO:teuthology.lock.ops:Updating [ubuntu@smithi038.front.sepia.ceph.com]: reset os type and version on server 2021-06-26T23:54:06.078 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2021-06-26T23:54:06.094 INFO:teuthology.lock.ops:Node 'ubuntu@smithi038.front.sepia.ceph.com' reimaging is complete 2021-06-26T23:54:06.284 INFO:teuthology.provision.fog.smithi022:Scheduling deploy of centos 8.3 2021-06-26T23:54:06.290 INFO:teuthology.provision.fog.smithi038:Scheduling deploy of centos 8.3 2021-06-26T23:54:06.677 INFO:teuthology.orchestra.console:Power off smithi038 2021-06-26T23:54:06.738 INFO:teuthology.orchestra.console:Power off smithi022 2021-06-26T23:54:15.143 INFO:teuthology.orchestra.console:Power off for smithi038 completed 2021-06-26T23:54:15.161 INFO:teuthology.orchestra.console:Power off for smithi022 completed 2021-06-26T23:54:15.245 INFO:teuthology.orchestra.console:Power on smithi038 2021-06-26T23:54:15.263 INFO:teuthology.orchestra.console:Power on smithi022 2021-06-26T23:54:23.638 INFO:teuthology.orchestra.console:Power on for smithi038 completed 2021-06-26T23:54:23.649 INFO:teuthology.orchestra.console:Power on for smithi022 completed 2021-06-26T23:54:23.740 INFO:teuthology.provision.fog.smithi038:Waiting for deploy to finish 2021-06-26T23:54:23.750 INFO:teuthology.provision.fog.smithi022:Waiting for deploy to finish 2021-06-26T23:58:40.866 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-26T23:58:40.954 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-26T23:58:41.164 INFO:teuthology.provision.fog.smithi038:Node is ready 2021-06-26T23:58:41.179 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038.front.sepia.ceph.com 2021-06-26T23:58:41.237 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38 smithi038.front.sepia.ceph.com smithi038 2021-06-26T23:58:41.278 INFO:teuthology.provision.fog.smithi022:Node is ready 2021-06-26T23:58:41.334 INFO:teuthology.orchestra.run.smithi022.stdout:smithi022.front.sepia.ceph.com 2021-06-26T23:58:41.389 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22 smithi022.front.sepia.ceph.com smithi022 2021-06-26T23:58:41.561 INFO:teuthology.provision.fog.smithi038:Deploy complete! 2021-06-26T23:58:41.722 INFO:teuthology.provision.fog.smithi022:Deploy complete! 2021-06-26T23:58:41.908 INFO:teuthology.lock.ops:Checking smithi022.front.sepia.ceph.com 2021-06-26T23:58:41.909 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-26T23:58:41.931 INFO:teuthology.lock.ops:Checking smithi038.front.sepia.ceph.com 2021-06-26T23:58:41.932 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-26T23:58:41.947 INFO:teuthology.lock.ops:Updating [smithi022.front.sepia.ceph.com]: set os type and version on server 2021-06-26T23:58:41.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-26T23:58:42.235 INFO:teuthology.orchestra.run.smithi022.stdout:x86_64 2021-06-26T23:58:42.290 INFO:teuthology.orchestra.run.smithi022.stdout:NAME="CentOS Linux" 2021-06-26T23:58:42.291 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION="8" 2021-06-26T23:58:42.291 INFO:teuthology.orchestra.run.smithi022.stdout:ID="centos" 2021-06-26T23:58:42.291 INFO:teuthology.orchestra.run.smithi022.stdout:ID_LIKE="rhel fedora" 2021-06-26T23:58:42.291 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION_ID="8" 2021-06-26T23:58:42.291 INFO:teuthology.orchestra.run.smithi022.stdout:PLATFORM_ID="platform:el8" 2021-06-26T23:58:42.292 INFO:teuthology.orchestra.run.smithi022.stdout:PRETTY_NAME="CentOS Linux 8" 2021-06-26T23:58:42.292 INFO:teuthology.orchestra.run.smithi022.stdout:ANSI_COLOR="0;31" 2021-06-26T23:58:42.292 INFO:teuthology.orchestra.run.smithi022.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-26T23:58:42.292 INFO:teuthology.orchestra.run.smithi022.stdout:HOME_URL="https://centos.org/" 2021-06-26T23:58:42.293 INFO:teuthology.orchestra.run.smithi022.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-26T23:58:42.293 INFO:teuthology.orchestra.run.smithi022.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-26T23:58:42.293 INFO:teuthology.orchestra.run.smithi022.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-26T23:58:42.298 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2021-06-26T23:58:42.316 INFO:teuthology.lock.ops:Updating [smithi038.front.sepia.ceph.com]: set os type and version on server 2021-06-26T23:58:42.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-26T23:58:42.703 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2021-06-26T23:58:42.762 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Linux" 2021-06-26T23:58:42.762 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="8" 2021-06-26T23:58:42.762 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2021-06-26T23:58:42.763 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2021-06-26T23:58:42.763 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="8" 2021-06-26T23:58:42.763 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el8" 2021-06-26T23:58:42.763 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Linux 8" 2021-06-26T23:58:42.764 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2021-06-26T23:58:42.764 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-26T23:58:42.764 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2021-06-26T23:58:42.764 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-26T23:58:42.765 INFO:teuthology.orchestra.run.smithi038.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-26T23:58:42.771 INFO:teuthology.orchestra.run.smithi038.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-26T23:58:42.771 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2021-06-26T23:58:43.001 INFO:teuthology.dispatcher.supervisor:Running job 6194740 2021-06-26T23:58:43.001 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-06-26_19:32:15-rados-wip-ronenf-scrubs-config-distro-basic-smithi/6194740 --name rfriedma-2021-06-26_19:32:15-rados-wip-ronenf-scrubs-config-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-agent-big} -- /home/teuthworker/archive/rfriedma-2021-06-26_19:32:15-rados-wip-ronenf-scrubs-config-distro-basic-smithi/6194740/orig.config.yaml 2021-06-26T23:58:43.129 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-06-26_19:32:15-rados-wip-ronenf-scrubs-config-distro-basic-smithi/6194740 2021-06-26T23:58:43.130 INFO:teuthology.dispatcher.supervisor:Job PID: 32017 2021-06-26T23:58:43.130 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-06-27T00:14:48.372 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/rfriedma-2021-06-26_19:32:15-rados-wip-ronenf-scrubs-config-distro-basic-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2021-06-27T12:00:30.155 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-06-27T12:00:30.281 INFO:teuthology.kill:Killing Pids: {32017} 2021-06-27T12:00:31.254 INFO:teuthology.kill:Nuking machines: ['smithi022', 'smithi038'] 2021-06-27T12:00:32.086 INFO:teuthology.kill:2021-06-27 12:00:32,085.085 INFO:teuthology.nuke:targets: 2021-06-27T12:00:32.087 INFO:teuthology.kill: smithi022.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/WtmMzizOw49GUFdPFH7koEEeVdX8GM/g8KnfeN6rv5qghrqnoFiI/HHE91HIVYhfx5G6QnA1d/APzmWn+mOE32mKnT0eAKxgIar6PF23UkMc82aAvwMSfW6Ml21opLuvmgFamVoemaPtPVoX2JRbw9AqvV+3cl+9yGeFmLD2uLF0B+Sg/j/JDulZ8sopPyFS2DjIKEPhwzf6Qvx6IZJKIU2+gkDrLcizt/ZrNkHKRdlO5n6y1bkpb+rGew3xdBWpY58C0PYkoR0kHuy5dF53Hx1VQN4ZGYNx4QMKit7O/z8SQ4OS+o+mpEBaQA+X5BA5q0UMIvhCdj7cAassJSYrbP0dS/Wi04uZgnIP3JG9dSY6DpfgYOJuMarLMI8ljuDN6JWOrru+/CHDnpE4kh+hghrhS9OKh6eJuXx73xu+fwIcXEkKUZAn44ll+j8RJH5UH9gjrR/Mu3wRtlndN1/hoqSvxA+I6RsTzyf5Tm3p0WB6/ew0xEJN8CKHn3KMh2c= 2021-06-27T12:00:32.087 INFO:teuthology.kill: smithi038.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMKHL2GnSqXm6TUCiT+4uCrmRRnp6gLqHUiM6FirkWCbFogtTFl+r+lm/EFFLrlEh3pG5NBzmpFHjqreWIC6GJzQREtqNXGQVMapAejfOWDh2rcjfz8IFJjxq9j5WE81GsPnOEdwOrQ6ZhwtH+js3FizFafCmu30InD7FTZyIUIBHgcX5wft1/FEMuYbtynUdSUsGkf2Q4WjzM/GnbhRH3NewfdV10oQU7vSQKpDCiscW8OzVWmU/L4yShB7tL9rknWh/BP9/06icT4LkJMzyLMAnXYGmRC/0j4vaa+YUwbNKLacyVFVEdLYbRbMjjqlyAeVOpA5nCL1m0YYk0rwBSQgLum+BgyhP5TS+Mj3kzofuLwbVIoAthraCbOaBeWoH31RyhuVyhwGUhrKF50JKcgI1SeuJXce17tV/u6taNnVHAJhHHi/OMa+QE+U9GAUNZzIgVcPTgmdyrfrObOzJNutM3D52ANDjWnw7BeRaiqRcqtpeYX4A3cwyBihnv+tk= 2021-06-27T12:00:32.088 INFO:teuthology.kill:2021-06-27 12:00:32,086.086 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-27T12:00:32.088 INFO:teuthology.kill:2021-06-27 12:00:32,088.088 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-27T12:00:32.144 INFO:teuthology.kill:2021-06-27 12:00:32,144.144 INFO:teuthology.orchestra.console:Power off smithi022 2021-06-27T12:00:32.157 INFO:teuthology.kill:2021-06-27 12:00:32,157.157 INFO:teuthology.orchestra.console:Power off smithi038 2021-06-27T12:00:40.517 INFO:teuthology.kill:2021-06-27 12:00:40,516.516 INFO:teuthology.orchestra.console:Power off for smithi022 completed 2021-06-27T12:00:44.631 INFO:teuthology.kill:2021-06-27 12:00:44,631.631 INFO:teuthology.orchestra.console:Power off for smithi038 completed 2021-06-27T12:00:44.857 INFO:teuthology.task.internal:roles: ubuntu@smithi022.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-06-27T12:00:44.857 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-06-27T12:00:44.858 INFO:teuthology.misc:Compressing logs... 2021-06-27T12:00:44.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-27T12:01:19.307 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-06-27T12:01:19.369 INFO:teuthology.kill:No teuthology processes running 2021-06-27T12:01:20.296 INFO:teuthology.kill:Nuking machines: ['smithi022', 'smithi038'] 2021-06-27T12:01:21.123 INFO:teuthology.kill:2021-06-27 12:01:21,123.123 INFO:teuthology.nuke:targets: 2021-06-27T12:01:21.124 INFO:teuthology.kill: smithi022.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/WtmMzizOw49GUFdPFH7koEEeVdX8GM/g8KnfeN6rv5qghrqnoFiI/HHE91HIVYhfx5G6QnA1d/APzmWn+mOE32mKnT0eAKxgIar6PF23UkMc82aAvwMSfW6Ml21opLuvmgFamVoemaPtPVoX2JRbw9AqvV+3cl+9yGeFmLD2uLF0B+Sg/j/JDulZ8sopPyFS2DjIKEPhwzf6Qvx6IZJKIU2+gkDrLcizt/ZrNkHKRdlO5n6y1bkpb+rGew3xdBWpY58C0PYkoR0kHuy5dF53Hx1VQN4ZGYNx4QMKit7O/z8SQ4OS+o+mpEBaQA+X5BA5q0UMIvhCdj7cAassJSYrbP0dS/Wi04uZgnIP3JG9dSY6DpfgYOJuMarLMI8ljuDN6JWOrru+/CHDnpE4kh+hghrhS9OKh6eJuXx73xu+fwIcXEkKUZAn44ll+j8RJH5UH9gjrR/Mu3wRtlndN1/hoqSvxA+I6RsTzyf5Tm3p0WB6/ew0xEJN8CKHn3KMh2c= 2021-06-27T12:01:21.124 INFO:teuthology.kill: smithi038.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMKHL2GnSqXm6TUCiT+4uCrmRRnp6gLqHUiM6FirkWCbFogtTFl+r+lm/EFFLrlEh3pG5NBzmpFHjqreWIC6GJzQREtqNXGQVMapAejfOWDh2rcjfz8IFJjxq9j5WE81GsPnOEdwOrQ6ZhwtH+js3FizFafCmu30InD7FTZyIUIBHgcX5wft1/FEMuYbtynUdSUsGkf2Q4WjzM/GnbhRH3NewfdV10oQU7vSQKpDCiscW8OzVWmU/L4yShB7tL9rknWh/BP9/06icT4LkJMzyLMAnXYGmRC/0j4vaa+YUwbNKLacyVFVEdLYbRbMjjqlyAeVOpA5nCL1m0YYk0rwBSQgLum+BgyhP5TS+Mj3kzofuLwbVIoAthraCbOaBeWoH31RyhuVyhwGUhrKF50JKcgI1SeuJXce17tV/u6taNnVHAJhHHi/OMa+QE+U9GAUNZzIgVcPTgmdyrfrObOzJNutM3D52ANDjWnw7BeRaiqRcqtpeYX4A3cwyBihnv+tk= 2021-06-27T12:01:21.143 INFO:teuthology.kill:2021-06-27 12:01:21,143.143 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-27T12:01:21.174 INFO:teuthology.kill:2021-06-27 12:01:21,174.174 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-27T12:01:21.192 INFO:teuthology.kill:2021-06-27 12:01:21,191.191 INFO:teuthology.orchestra.console:Power off smithi022 2021-06-27T12:01:21.238 INFO:teuthology.kill:2021-06-27 12:01:21,237.237 INFO:teuthology.orchestra.console:Power off smithi038 2021-06-27T12:01:21.332 INFO:teuthology.kill:2021-06-27 12:01:21,332.332 INFO:teuthology.orchestra.console:Power off for smithi022 completed 2021-06-27T12:01:21.377 INFO:teuthology.kill:2021-06-27 12:01:21,377.377 INFO:teuthology.orchestra.console:Power off for smithi038 completed 2021-06-27T12:01:21.467 INFO:teuthology.kill:2021-06-27 12:01:21,466.466 INFO:teuthology.lock.ops:unlocked smithi022.front.sepia.ceph.com 2021-06-27T12:01:21.511 INFO:teuthology.kill:2021-06-27 12:01:21,511.511 INFO:teuthology.lock.ops:unlocked smithi038.front.sepia.ceph.com 2021-06-27T12:03:21.930 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15