2021-04-24T14:25:56.183 INFO:root:teuthology version: 1.1.0-2713a3cd
2021-04-24T14:25:56.282 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba009.front.sepia.ceph.com' reimaging
2021-04-24T14:25:56.282 INFO:teuthology.lock.ops:Updating [ubuntu@gibba009.front.sepia.ceph.com]: reset os type and version on server
2021-04-24T14:25:56.282 INFO:teuthology.lock.ops:Updating gibba009.front.sepia.ceph.com on lock server
2021-04-24T14:25:56.354 INFO:teuthology.lock.ops:Node 'ubuntu@gibba009.front.sepia.ceph.com' reimaging is complete
2021-04-24T14:25:56.354 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba043.front.sepia.ceph.com' reimaging
2021-04-24T14:25:56.354 INFO:teuthology.lock.ops:Updating [ubuntu@gibba043.front.sepia.ceph.com]: reset os type and version on server
2021-04-24T14:25:56.354 INFO:teuthology.lock.ops:Updating gibba043.front.sepia.ceph.com on lock server
2021-04-24T14:25:56.402 INFO:teuthology.lock.ops:Node 'ubuntu@gibba043.front.sepia.ceph.com' reimaging is complete
2021-04-24T14:25:56.550 INFO:teuthology.provision.fog.gibba009:Scheduling deploy of centos 8.3
2021-04-24T14:25:56.603 INFO:teuthology.provision.fog.gibba043:Scheduling deploy of centos 8.3
2021-04-24T14:25:56.828 INFO:teuthology.orchestra.console:Power off gibba009
2021-04-24T14:25:56.878 INFO:teuthology.orchestra.console:Power off gibba043
2021-04-24T14:25:56.973 INFO:teuthology.orchestra.console:Power off for gibba009 completed
2021-04-24T14:25:57.040 INFO:teuthology.orchestra.console:Power off for gibba043 completed
2021-04-24T14:25:57.074 INFO:teuthology.orchestra.console:Power on gibba009
2021-04-24T14:25:57.140 INFO:teuthology.orchestra.console:Power on gibba043
2021-04-24T14:26:01.339 INFO:teuthology.orchestra.console:Power on for gibba009 completed
2021-04-24T14:26:01.440 INFO:teuthology.provision.fog.gibba009:Waiting for deploy to finish
2021-04-24T14:26:05.513 INFO:teuthology.orchestra.console:Power on for gibba043 completed
2021-04-24T14:26:05.614 INFO:teuthology.provision.fog.gibba043:Waiting for deploy to finish
2021-04-24T14:30:05.753 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-04-24T14:30:16.489 INFO:teuthology.provision.fog.gibba043:Node is ready
2021-04-24T14:30:16.502 INFO:teuthology.orchestra.run.gibba043.stdout:gibba029
2021-04-24T14:30:16.554 INFO:teuthology.orchestra.run.gibba043.stdout:172.21.2.129 gibba029.front.sepia.ceph.com gibba029
2021-04-24T14:30:16.858 INFO:teuthology.provision.fog.gibba043:Deploy complete!
2021-04-24T14:30:27.094 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-04-24T14:30:27.388 INFO:teuthology.provision.fog.gibba009:Node is ready
2021-04-24T14:30:27.440 INFO:teuthology.orchestra.run.gibba009.stdout:gibba009.front.sepia.ceph.com
2021-04-24T14:30:27.492 INFO:teuthology.orchestra.run.gibba009.stdout:172.21.2.109 gibba009.front.sepia.ceph.com gibba009
2021-04-24T14:30:27.776 INFO:teuthology.provision.fog.gibba009:Deploy complete!
2021-04-24T14:30:27.912 INFO:teuthology.lock.ops:Checking gibba009.front.sepia.ceph.com
2021-04-24T14:30:27.913 INFO:teuthology.lock.ops:New key found. Updating...
2021-04-24T14:30:27.929 INFO:teuthology.lock.ops:Checking gibba043.front.sepia.ceph.com
2021-04-24T14:30:27.929 INFO:teuthology.lock.ops:New key found. Updating...
2021-04-24T14:30:27.944 INFO:teuthology.lock.ops:Updating [gibba009.front.sepia.ceph.com]: set os type and version on server
2021-04-24T14:30:27.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-24T14:30:28.186 INFO:teuthology.orchestra.run.gibba009.stdout:x86_64
2021-04-24T14:30:28.237 INFO:teuthology.orchestra.run.gibba009.stdout:NAME="CentOS Linux"
2021-04-24T14:30:28.237 INFO:teuthology.orchestra.run.gibba009.stdout:VERSION="8"
2021-04-24T14:30:28.237 INFO:teuthology.orchestra.run.gibba009.stdout:ID="centos"
2021-04-24T14:30:28.237 INFO:teuthology.orchestra.run.gibba009.stdout:ID_LIKE="rhel fedora"
2021-04-24T14:30:28.237 INFO:teuthology.orchestra.run.gibba009.stdout:VERSION_ID="8"
2021-04-24T14:30:28.237 INFO:teuthology.orchestra.run.gibba009.stdout:PLATFORM_ID="platform:el8"
2021-04-24T14:30:28.237 INFO:teuthology.orchestra.run.gibba009.stdout:PRETTY_NAME="CentOS Linux 8"
2021-04-24T14:30:28.238 INFO:teuthology.orchestra.run.gibba009.stdout:ANSI_COLOR="0;31"
2021-04-24T14:30:28.238 INFO:teuthology.orchestra.run.gibba009.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-04-24T14:30:28.238 INFO:teuthology.orchestra.run.gibba009.stdout:HOME_URL="https://centos.org/"
2021-04-24T14:30:28.238 INFO:teuthology.orchestra.run.gibba009.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-04-24T14:30:28.238 INFO:teuthology.orchestra.run.gibba009.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-04-24T14:30:28.238 INFO:teuthology.orchestra.run.gibba009.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-04-24T14:30:28.239 INFO:teuthology.lock.ops:Updating gibba009.front.sepia.ceph.com on lock server
2021-04-24T14:30:28.255 INFO:teuthology.lock.ops:Updating [gibba043.front.sepia.ceph.com]: set os type and version on server
2021-04-24T14:30:28.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-24T14:30:28.503 INFO:teuthology.orchestra.run.gibba043.stdout:x86_64
2021-04-24T14:30:28.555 INFO:teuthology.orchestra.run.gibba043.stdout:NAME="CentOS Linux"
2021-04-24T14:30:28.556 INFO:teuthology.orchestra.run.gibba043.stdout:VERSION="8"
2021-04-24T14:30:28.556 INFO:teuthology.orchestra.run.gibba043.stdout:ID="centos"
2021-04-24T14:30:28.556 INFO:teuthology.orchestra.run.gibba043.stdout:ID_LIKE="rhel fedora"
2021-04-24T14:30:28.556 INFO:teuthology.orchestra.run.gibba043.stdout:VERSION_ID="8"
2021-04-24T14:30:28.556 INFO:teuthology.orchestra.run.gibba043.stdout:PLATFORM_ID="platform:el8"
2021-04-24T14:30:28.556 INFO:teuthology.orchestra.run.gibba043.stdout:PRETTY_NAME="CentOS Linux 8"
2021-04-24T14:30:28.556 INFO:teuthology.orchestra.run.gibba043.stdout:ANSI_COLOR="0;31"
2021-04-24T14:30:28.556 INFO:teuthology.orchestra.run.gibba043.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-04-24T14:30:28.556 INFO:teuthology.orchestra.run.gibba043.stdout:HOME_URL="https://centos.org/"
2021-04-24T14:30:28.556 INFO:teuthology.orchestra.run.gibba043.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-04-24T14:30:28.556 INFO:teuthology.orchestra.run.gibba043.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-04-24T14:30:28.557 INFO:teuthology.orchestra.run.gibba043.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-04-24T14:30:28.557 INFO:teuthology.lock.ops:Updating gibba043.front.sepia.ceph.com on lock server
2021-04-24T14:30:28.653 INFO:teuthology.dispatcher.supervisor:Running job 6069497
2021-04-24T14:30:28.653 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-04-23_22:57:33-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6069497 --name rfriedma-2021-04-23_22:57:33-rados-wip-ronenf-scrub-sched-distro-basic-gibba --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/radosbench} -- /home/teuthworker/archive/rfriedma-2021-04-23_22:57:33-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6069497/orig.config.yaml
2021-04-24T14:30:28.659 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-04-23_22:57:33-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6069497
2021-04-24T14:30:28.659 INFO:teuthology.dispatcher.supervisor:Job PID: 23631
2021-04-24T14:30:28.659 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-04-24T14:50:35.393 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/rfriedma-2021-04-23_22:57:33-rados-wip-ronenf-scrub-sched-distro-basic-gibba/jobs/ failed with status 504:
504 Gateway Time-out
504 Gateway Time-out
nginx/1.10.3 (Ubuntu)
2021-04-25T02:31:08.094 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-04-25T02:31:08.291 INFO:teuthology.kill:Killing Pids: {23631}
2021-04-25T02:31:09.314 INFO:teuthology.kill:Nuking machines: ['gibba009', 'gibba043']
2021-04-25T02:31:10.248 INFO:teuthology.kill:2021-04-25 02:31:10,248.248 INFO:teuthology.nuke:targets:
2021-04-25T02:31:10.248 INFO:teuthology.kill: gibba009.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnnhrl4wDkGXSVOTGxEMmk+3MlGfYfJSYb4yBETevns10/UAH1dAjfIbozRx2Ew7Mu4WxGIAa+rruipkdCc698fJ+7R/2L0O6npBPf0Xl0YwkpmohZ5fpKNlQO7NctjBS/MgYkU2yyPEZQ+B+1AX9T91HvadIUzphmzIG+hDLzcxN7ZtXipGeHnCcuin4sz6lGi7qtptEG1p/SddocASLEY/pbTSAFGWXadODVPzRibFfdzbjGJbz+xbHxZCzC89aVpCj1Tz2M2A2YduHEfPUVIErp1EU99BDfhK/pIRgWhIlP9mSC95s4u6PDjP4Tbw+ML/Qj8IsTRX7V9gBlnsW+cLcB2DYEA9ou84CmErJvC8OI9jdrdXthJJm+p7RwrEbfdIMGmQgH3LvLfseIwNqOAmAOKmElR6ObLRL9BbQuvXnSpBSkTBqAlaHjHmFsls+ADYCGWGBdA7XV59SWerKzvsr+t49WsiUvb2g6C3mUMIw5/g9CBRU2p6Uj9KxfaSk=
2021-04-25T02:31:10.248 INFO:teuthology.kill: gibba043.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0DjdSZ8KgcymtjwzgehQ92OzE2iB2/75zv3/r8h5RMXNsyv1Kg9c2NAxOnzyLdb6at2ZLqu+NmKZXIKSkhvzRLyXseU83vgOwhu2nETZjFJTVPPKYxS59JqAiGO1yWFiSB37LlrZw80x1jPbT5Qlefo0CiHX/b09GmlAqMoExj7YxVTNvrle+PNXtAi3RgVZuw6CWoOOQSM6HcjCrIgWrsboUVCoJIOPvIwoeM42FCycWiRTJOvNyLxDvIA9dS8a7Qd12SxTk7QqI3EsBLLOqmvD2r7tpBBKlZGkpbtJSkh/JSQZukX2q5wKbTLSsSGw5TtjL8c15/aH+b0WUbStCzxhcTGQ17EpBlDXUca7c9ActN8giE/kEgdKsGDSRDvCKvr1/Hg8TL8YD84q0oXLSAbu0DHDqHR7i+6F7Bv9bYJH7p7xd1wJ2L6yNIQwhYAPvkdfg4t9dogasCo0Aj3K52eV9iCkt5Wfl8FkvVnCrnOswq2c7T+QDelV7AdGmcRc=
2021-04-25T02:31:10.249 INFO:teuthology.kill:2021-04-25 02:31:10,248.248 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-25T02:31:10.250 INFO:teuthology.kill:2021-04-25 02:31:10,250.250 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-25T02:31:10.341 INFO:teuthology.kill:2021-04-25 02:31:10,341.341 INFO:teuthology.orchestra.console:Power off gibba009
2021-04-25T02:31:10.356 INFO:teuthology.kill:2021-04-25 02:31:10,356.356 INFO:teuthology.orchestra.console:Power off gibba043
2021-04-25T02:31:18.727 INFO:teuthology.kill:2021-04-25 02:31:18,726.726 INFO:teuthology.orchestra.console:Power off for gibba009 completed
2021-04-25T02:31:18.801 INFO:teuthology.kill:2021-04-25 02:31:18,801.801 INFO:teuthology.orchestra.console:Power off for gibba043 completed
2021-04-25T02:31:19.034 INFO:teuthology.task.internal:roles: ubuntu@gibba009.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2021-04-25T02:31:19.034 INFO:teuthology.task.internal:roles: ubuntu@gibba043.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2021-04-25T02:31:19.034 INFO:teuthology.misc:Compressing logs...
2021-04-25T02:31:19.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-25T02:32:19.095 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
teuth_config.archive_base, job_config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
compress_logs(ctx, log_path)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/misc.py", line 1378, in compress_logs
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/remote.py", line 215, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 434, in run
transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-04-25T02:32:19.164 INFO:teuthology.kill:No teuthology processes running
2021-04-25T02:32:20.256 INFO:teuthology.kill:Nuking machines: ['gibba009', 'gibba043']
2021-04-25T02:32:21.190 INFO:teuthology.kill:2021-04-25 02:32:21,190.190 INFO:teuthology.nuke:targets:
2021-04-25T02:32:21.191 INFO:teuthology.kill: gibba009.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnnhrl4wDkGXSVOTGxEMmk+3MlGfYfJSYb4yBETevns10/UAH1dAjfIbozRx2Ew7Mu4WxGIAa+rruipkdCc698fJ+7R/2L0O6npBPf0Xl0YwkpmohZ5fpKNlQO7NctjBS/MgYkU2yyPEZQ+B+1AX9T91HvadIUzphmzIG+hDLzcxN7ZtXipGeHnCcuin4sz6lGi7qtptEG1p/SddocASLEY/pbTSAFGWXadODVPzRibFfdzbjGJbz+xbHxZCzC89aVpCj1Tz2M2A2YduHEfPUVIErp1EU99BDfhK/pIRgWhIlP9mSC95s4u6PDjP4Tbw+ML/Qj8IsTRX7V9gBlnsW+cLcB2DYEA9ou84CmErJvC8OI9jdrdXthJJm+p7RwrEbfdIMGmQgH3LvLfseIwNqOAmAOKmElR6ObLRL9BbQuvXnSpBSkTBqAlaHjHmFsls+ADYCGWGBdA7XV59SWerKzvsr+t49WsiUvb2g6C3mUMIw5/g9CBRU2p6Uj9KxfaSk=
2021-04-25T02:32:21.191 INFO:teuthology.kill: gibba043.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0DjdSZ8KgcymtjwzgehQ92OzE2iB2/75zv3/r8h5RMXNsyv1Kg9c2NAxOnzyLdb6at2ZLqu+NmKZXIKSkhvzRLyXseU83vgOwhu2nETZjFJTVPPKYxS59JqAiGO1yWFiSB37LlrZw80x1jPbT5Qlefo0CiHX/b09GmlAqMoExj7YxVTNvrle+PNXtAi3RgVZuw6CWoOOQSM6HcjCrIgWrsboUVCoJIOPvIwoeM42FCycWiRTJOvNyLxDvIA9dS8a7Qd12SxTk7QqI3EsBLLOqmvD2r7tpBBKlZGkpbtJSkh/JSQZukX2q5wKbTLSsSGw5TtjL8c15/aH+b0WUbStCzxhcTGQ17EpBlDXUca7c9ActN8giE/kEgdKsGDSRDvCKvr1/Hg8TL8YD84q0oXLSAbu0DHDqHR7i+6F7Bv9bYJH7p7xd1wJ2L6yNIQwhYAPvkdfg4t9dogasCo0Aj3K52eV9iCkt5Wfl8FkvVnCrnOswq2c7T+QDelV7AdGmcRc=
2021-04-25T02:32:21.218 INFO:teuthology.kill:2021-04-25 02:32:21,218.218 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-25T02:32:21.236 INFO:teuthology.kill:2021-04-25 02:32:21,236.236 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-25T02:32:21.364 INFO:teuthology.kill:2021-04-25 02:32:21,364.364 INFO:teuthology.orchestra.console:Power off gibba009
2021-04-25T02:32:21.376 INFO:teuthology.kill:2021-04-25 02:32:21,376.376 INFO:teuthology.orchestra.console:Power off gibba043
2021-04-25T02:32:21.509 INFO:teuthology.kill:2021-04-25 02:32:21,509.509 INFO:teuthology.orchestra.console:Power off for gibba009 completed
2021-04-25T02:32:21.523 INFO:teuthology.kill:2021-04-25 02:32:21,523.523 INFO:teuthology.orchestra.console:Power off for gibba043 completed
2021-04-25T02:32:21.659 INFO:teuthology.kill:2021-04-25 02:32:21,659.659 INFO:teuthology.lock.ops:unlocked gibba009.front.sepia.ceph.com
2021-04-25T02:32:21.677 INFO:teuthology.kill:2021-04-25 02:32:21,676.676 INFO:teuthology.lock.ops:unlocked gibba043.front.sepia.ceph.com
2021-04-25T02:34:21.943 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-04-25T02:34:22.043 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-04-25T02:34:22.043 INFO:teuthology.nuke:Checking targets against current locks
2021-04-25T02:34:22.108 INFO:teuthology.nuke:Not nuking gibba043.front.sepia.ceph.com because description doesn't match
2021-04-25T02:34:22.109 INFO:teuthology.nuke:Not nuking gibba009.front.sepia.ceph.com because description doesn't match