2021-05-21T14:47:01.721 INFO:root:teuthology version: 1.1.0-471a2f1e 2021-05-21T14:47:01.936 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba005.front.sepia.ceph.com' reimaging 2021-05-21T14:47:01.937 INFO:teuthology.lock.ops:Updating [ubuntu@gibba005.front.sepia.ceph.com]: reset os type and version on server 2021-05-21T14:47:01.937 INFO:teuthology.lock.ops:Updating gibba005.front.sepia.ceph.com on lock server 2021-05-21T14:47:01.953 INFO:teuthology.lock.ops:Node 'ubuntu@gibba005.front.sepia.ceph.com' reimaging is complete 2021-05-21T14:47:01.953 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba014.front.sepia.ceph.com' reimaging 2021-05-21T14:47:01.953 INFO:teuthology.lock.ops:Updating [ubuntu@gibba014.front.sepia.ceph.com]: reset os type and version on server 2021-05-21T14:47:01.954 INFO:teuthology.lock.ops:Updating gibba014.front.sepia.ceph.com on lock server 2021-05-21T14:47:01.968 INFO:teuthology.lock.ops:Node 'ubuntu@gibba014.front.sepia.ceph.com' reimaging is complete 2021-05-21T14:47:02.159 INFO:teuthology.provision.fog.gibba005:Scheduling deploy of ubuntu 20.04 2021-05-21T14:47:02.177 INFO:teuthology.provision.fog.gibba014:Scheduling deploy of ubuntu 20.04 2021-05-21T14:47:02.557 INFO:teuthology.orchestra.console:Power off gibba014 2021-05-21T14:47:02.640 INFO:teuthology.orchestra.console:Power off gibba005 2021-05-21T14:47:11.067 INFO:teuthology.orchestra.console:Power off for gibba014 completed 2021-05-21T14:47:11.175 INFO:teuthology.orchestra.console:Power on gibba014 2021-05-21T14:47:15.247 INFO:teuthology.orchestra.console:Power off for gibba005 completed 2021-05-21T14:47:15.357 INFO:teuthology.orchestra.console:Power on gibba005 2021-05-21T14:47:19.670 INFO:teuthology.orchestra.console:Power on for gibba014 completed 2021-05-21T14:47:19.693 INFO:teuthology.orchestra.console:Power on for gibba005 completed 2021-05-21T14:47:19.772 INFO:teuthology.provision.fog.gibba014:Waiting for deploy to finish 2021-05-21T14:47:19.795 INFO:teuthology.provision.fog.gibba005:Waiting for deploy to finish 2021-05-21T14:49:51.068 ERROR:teuthology.orchestra.connection:Error authenticating with gibba005.front.sepia.ceph.com: Authentication failed. 2021-05-21T14:52:02.217 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-21T14:52:02.559 INFO:teuthology.provision.fog.gibba014:Node is ready 2021-05-21T14:52:02.563 INFO:teuthology.orchestra.run.gibba014.stdout:gibba014.front.sepia.ceph.com 2021-05-21T14:52:02.570 INFO:teuthology.orchestra.run.gibba014.stdout:172.21.2.114 gibba014.front.sepia.ceph.com gibba014 2021-05-21T14:52:02.798 INFO:teuthology.provision.fog.gibba014:Deploy complete! 2021-05-21T14:52:03.746 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-21T14:52:04.079 INFO:teuthology.provision.fog.gibba005:Node is ready 2021-05-21T14:52:04.084 INFO:teuthology.orchestra.run.gibba005.stdout:gibba005.front.sepia.ceph.com 2021-05-21T14:52:04.092 INFO:teuthology.orchestra.run.gibba005.stdout:172.21.2.105 gibba005.front.sepia.ceph.com gibba005 2021-05-21T14:52:04.312 INFO:teuthology.provision.fog.gibba005:Deploy complete! 2021-05-21T14:52:04.542 INFO:teuthology.lock.ops:Checking gibba005.front.sepia.ceph.com 2021-05-21T14:52:04.543 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-21T14:52:04.560 INFO:teuthology.lock.ops:Checking gibba014.front.sepia.ceph.com 2021-05-21T14:52:04.561 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-21T14:52:04.574 INFO:teuthology.lock.ops:Updating [gibba005.front.sepia.ceph.com]: set os type and version on server 2021-05-21T14:52:04.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-21T14:52:04.759 INFO:teuthology.orchestra.run.gibba005.stdout:x86_64 2021-05-21T14:52:04.767 INFO:teuthology.orchestra.run.gibba005.stdout:NAME="Ubuntu" 2021-05-21T14:52:04.768 INFO:teuthology.orchestra.run.gibba005.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-21T14:52:04.768 INFO:teuthology.orchestra.run.gibba005.stdout:ID=ubuntu 2021-05-21T14:52:04.768 INFO:teuthology.orchestra.run.gibba005.stdout:ID_LIKE=debian 2021-05-21T14:52:04.769 INFO:teuthology.orchestra.run.gibba005.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-21T14:52:04.769 INFO:teuthology.orchestra.run.gibba005.stdout:VERSION_ID="20.04" 2021-05-21T14:52:04.769 INFO:teuthology.orchestra.run.gibba005.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-21T14:52:04.769 INFO:teuthology.orchestra.run.gibba005.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-21T14:52:04.769 INFO:teuthology.orchestra.run.gibba005.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-21T14:52:04.770 INFO:teuthology.orchestra.run.gibba005.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-21T14:52:04.770 INFO:teuthology.orchestra.run.gibba005.stdout:VERSION_CODENAME=focal 2021-05-21T14:52:04.770 INFO:teuthology.orchestra.run.gibba005.stdout:UBUNTU_CODENAME=focal 2021-05-21T14:52:04.772 INFO:teuthology.lock.ops:Updating gibba005.front.sepia.ceph.com on lock server 2021-05-21T14:52:04.787 INFO:teuthology.lock.ops:Updating [gibba014.front.sepia.ceph.com]: set os type and version on server 2021-05-21T14:52:04.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-21T14:52:04.967 INFO:teuthology.orchestra.run.gibba014.stdout:x86_64 2021-05-21T14:52:04.975 INFO:teuthology.orchestra.run.gibba014.stdout:NAME="Ubuntu" 2021-05-21T14:52:04.976 INFO:teuthology.orchestra.run.gibba014.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-21T14:52:04.976 INFO:teuthology.orchestra.run.gibba014.stdout:ID=ubuntu 2021-05-21T14:52:04.976 INFO:teuthology.orchestra.run.gibba014.stdout:ID_LIKE=debian 2021-05-21T14:52:04.976 INFO:teuthology.orchestra.run.gibba014.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-21T14:52:04.976 INFO:teuthology.orchestra.run.gibba014.stdout:VERSION_ID="20.04" 2021-05-21T14:52:04.977 INFO:teuthology.orchestra.run.gibba014.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-21T14:52:04.977 INFO:teuthology.orchestra.run.gibba014.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-21T14:52:04.977 INFO:teuthology.orchestra.run.gibba014.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-21T14:52:04.977 INFO:teuthology.orchestra.run.gibba014.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-21T14:52:04.977 INFO:teuthology.orchestra.run.gibba014.stdout:VERSION_CODENAME=focal 2021-05-21T14:52:04.978 INFO:teuthology.orchestra.run.gibba014.stdout:UBUNTU_CODENAME=focal 2021-05-21T14:52:04.978 INFO:teuthology.lock.ops:Updating gibba014.front.sepia.ceph.com on lock server 2021-05-21T14:52:05.115 INFO:teuthology.dispatcher.supervisor:Running job 6126460 2021-05-21T14:52:05.116 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-05-21_12:26:30-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6126460 --name rfriedma-2021-05-21_12:26:30-rados-wip-ronenf-scrub-sched-distro-basic-gibba --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps-readproxy} -- /home/teuthworker/archive/rfriedma-2021-05-21_12:26:30-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6126460/orig.config.yaml 2021-05-21T14:52:05.119 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-05-21_12:26:30-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6126460 2021-05-21T14:52:05.120 INFO:teuthology.dispatcher.supervisor:Job PID: 24788 2021-05-21T14:52:05.120 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-21T23:56:19.445 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/rfriedma-2021-05-21_12:26:30-rados-wip-ronenf-scrub-sched-distro-basic-gibba/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2021-05-22T02:52:24.304 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-22T02:52:24.565 INFO:teuthology.kill:Killing Pids: {24788} 2021-05-22T02:52:26.358 INFO:teuthology.kill:Nuking machines: ['gibba005', 'gibba014'] 2021-05-22T02:52:27.331 INFO:teuthology.kill:2021-05-22 02:52:27,331.331 INFO:teuthology.nuke:targets: 2021-05-22T02:52:27.332 INFO:teuthology.kill: gibba005.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFNr4aIFXF+CP5GizzhwVm9ozp8u6Te527BfzjP2sZJ+7l0vmebLXq+9YZd/kSsyysjf45o0XvC9bHNCjez4PALXHZNdopuUP1s/2upUQc+/afcsXKfRiDHt/7sMED0a0DZXcLpKsUyb4V7XaKLh89pJbgblkq5SbHzjPPKJZ3Vr7K0I8BbTB265KzcZzs/FBEKTCVpQcAUxz9x1DMndxUU3OwN9xSaQNbORr8o+ROvCAu/dHeb6SRILH9dwkC7ZuIPPyTC1gLV+hhHEW+QAJs86l+KErnp4WSbp88wbr0FZETVVscOJalxRD/ch2mnkOeyy4tYR7mqRLCwpYMK0QmUFSeu0SH1qVGGj/AiJqQuer3yrIjBUJwcFgQAPXcRJTvPiLSFp5kFBBENO/HJQnWkP21zzXucaFO8jfd4WtaPl99oTlI/lu4xxyCshgpSw6M2BgvEZXVANSluOAqRIfRnnxiPkGOaTzklyb1fiSKuVagx5TqEM58s3KVUPsPm58= 2021-05-22T02:52:27.333 INFO:teuthology.kill: gibba014.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEzqVb31k6nM4qu7WuU3cpmZMhRsQ70zePPAVFh90hEYadBJGaSKN+DzkFmp3paqGVebzjRDwh3jQnyoAyHNtDH4QLCSiQlELayPXXVhYBvK/6jimxMrde8fPfeGWrrnLeMn5xmKWWrJoe5Fe6ED/KdFaz1DIOEQOUXbl0K7CGlXcQ3gLaVhtsUHnU6N5lDLhFlhbOo1B+m7LC6u5qFF3vylX6CsLkVLJZh9uvzUlfkieMPrDLUaTnislE+0wSU5jDO8LlkEHTG2kDZ6qDcQKEFQuVGQukZZR1ikgj2l7N/Tzdpc35fheq1gvuWwp7ANByj032RA+XoBbwPqCwUxqf1HJo7E79/iFRKErJO2Gf6vyPlv6aYiC7k7bTcMAKMAqA6+Rw0xGHdh+ZkMfd1ZqshqJ1JYvZKLdoTYn9DFUphRAfgKHdkBiE25kpxZ7jlUMEqjZ9CZVb1Rc8xqs5R7DNj8YxpFCNCwPoOZ1r4FkvHyVxQTeWuL/yqDlSCumLyEU= 2021-05-22T02:52:27.333 INFO:teuthology.kill:2021-05-22 02:52:27,332.332 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-22T02:52:27.334 INFO:teuthology.kill:2021-05-22 02:52:27,334.334 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-22T02:52:27.395 INFO:teuthology.kill:2021-05-22 02:52:27,394.394 INFO:teuthology.orchestra.console:Power off gibba014 2021-05-22T02:52:27.409 INFO:teuthology.kill:2021-05-22 02:52:27,409.409 INFO:teuthology.orchestra.console:Power off gibba005 2021-05-22T02:52:35.799 INFO:teuthology.kill:2021-05-22 02:52:35,799.799 INFO:teuthology.orchestra.console:Power off for gibba005 completed 2021-05-22T02:52:39.888 INFO:teuthology.kill:2021-05-22 02:52:39,887.887 INFO:teuthology.orchestra.console:Power off for gibba014 completed 2021-05-22T02:52:40.127 INFO:teuthology.task.internal:roles: ubuntu@gibba005.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-05-22T02:52:40.127 INFO:teuthology.task.internal:roles: ubuntu@gibba014.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-05-22T02:52:40.128 INFO:teuthology.misc:Compressing logs... 2021-05-22T02:52:40.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-22T02:53:40.132 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-22T02:53:40.194 INFO:teuthology.kill:No teuthology processes running 2021-05-22T02:53:41.481 INFO:teuthology.kill:Nuking machines: ['gibba005', 'gibba014'] 2021-05-22T02:53:42.419 INFO:teuthology.kill:2021-05-22 02:53:42,419.419 INFO:teuthology.nuke:targets: 2021-05-22T02:53:42.420 INFO:teuthology.kill: gibba005.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFNr4aIFXF+CP5GizzhwVm9ozp8u6Te527BfzjP2sZJ+7l0vmebLXq+9YZd/kSsyysjf45o0XvC9bHNCjez4PALXHZNdopuUP1s/2upUQc+/afcsXKfRiDHt/7sMED0a0DZXcLpKsUyb4V7XaKLh89pJbgblkq5SbHzjPPKJZ3Vr7K0I8BbTB265KzcZzs/FBEKTCVpQcAUxz9x1DMndxUU3OwN9xSaQNbORr8o+ROvCAu/dHeb6SRILH9dwkC7ZuIPPyTC1gLV+hhHEW+QAJs86l+KErnp4WSbp88wbr0FZETVVscOJalxRD/ch2mnkOeyy4tYR7mqRLCwpYMK0QmUFSeu0SH1qVGGj/AiJqQuer3yrIjBUJwcFgQAPXcRJTvPiLSFp5kFBBENO/HJQnWkP21zzXucaFO8jfd4WtaPl99oTlI/lu4xxyCshgpSw6M2BgvEZXVANSluOAqRIfRnnxiPkGOaTzklyb1fiSKuVagx5TqEM58s3KVUPsPm58= 2021-05-22T02:53:42.420 INFO:teuthology.kill: gibba014.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEzqVb31k6nM4qu7WuU3cpmZMhRsQ70zePPAVFh90hEYadBJGaSKN+DzkFmp3paqGVebzjRDwh3jQnyoAyHNtDH4QLCSiQlELayPXXVhYBvK/6jimxMrde8fPfeGWrrnLeMn5xmKWWrJoe5Fe6ED/KdFaz1DIOEQOUXbl0K7CGlXcQ3gLaVhtsUHnU6N5lDLhFlhbOo1B+m7LC6u5qFF3vylX6CsLkVLJZh9uvzUlfkieMPrDLUaTnislE+0wSU5jDO8LlkEHTG2kDZ6qDcQKEFQuVGQukZZR1ikgj2l7N/Tzdpc35fheq1gvuWwp7ANByj032RA+XoBbwPqCwUxqf1HJo7E79/iFRKErJO2Gf6vyPlv6aYiC7k7bTcMAKMAqA6+Rw0xGHdh+ZkMfd1ZqshqJ1JYvZKLdoTYn9DFUphRAfgKHdkBiE25kpxZ7jlUMEqjZ9CZVb1Rc8xqs5R7DNj8YxpFCNCwPoOZ1r4FkvHyVxQTeWuL/yqDlSCumLyEU= 2021-05-22T02:53:42.440 INFO:teuthology.kill:2021-05-22 02:53:42,439.439 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-22T02:53:42.444 INFO:teuthology.kill:2021-05-22 02:53:42,443.443 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-22T02:53:42.496 INFO:teuthology.kill:2021-05-22 02:53:42,496.496 INFO:teuthology.orchestra.console:Power off gibba014 2021-05-22T02:53:42.510 INFO:teuthology.kill:2021-05-22 02:53:42,510.510 INFO:teuthology.orchestra.console:Power off gibba005 2021-05-22T02:53:42.643 INFO:teuthology.kill:2021-05-22 02:53:42,642.642 INFO:teuthology.orchestra.console:Power off for gibba014 completed 2021-05-22T02:53:42.658 INFO:teuthology.kill:2021-05-22 02:53:42,658.658 INFO:teuthology.orchestra.console:Power off for gibba005 completed 2021-05-22T02:53:42.787 INFO:teuthology.kill:2021-05-22 02:53:42,786.786 INFO:teuthology.lock.ops:unlocked gibba014.front.sepia.ceph.com 2021-05-22T02:53:42.805 INFO:teuthology.kill:2021-05-22 02:53:42,805.805 INFO:teuthology.lock.ops:unlocked gibba005.front.sepia.ceph.com 2021-05-22T02:55:42.985 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-05-22T02:55:43.081 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-05-22T02:55:43.082 INFO:teuthology.nuke:Checking targets against current locks 2021-05-22T02:55:43.179 INFO:teuthology.nuke:Not nuking gibba005.front.sepia.ceph.com because description doesn't match 2021-05-22T02:55:43.180 INFO:teuthology.nuke:Not nuking gibba014.front.sepia.ceph.com because description doesn't match