2021-04-23T17:42:31.748 INFO:root:teuthology version: 1.1.0-2713a3cd
2021-04-23T17:42:31.853 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba015.front.sepia.ceph.com' reimaging
2021-04-23T17:42:31.854 INFO:teuthology.lock.ops:Updating [ubuntu@gibba015.front.sepia.ceph.com]: reset os type and version on server
2021-04-23T17:42:31.854 INFO:teuthology.lock.ops:Updating gibba015.front.sepia.ceph.com on lock server
2021-04-23T17:42:31.870 INFO:teuthology.lock.ops:Node 'ubuntu@gibba015.front.sepia.ceph.com' reimaging is complete
2021-04-23T17:42:31.870 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba022.front.sepia.ceph.com' reimaging
2021-04-23T17:42:31.871 INFO:teuthology.lock.ops:Updating [ubuntu@gibba022.front.sepia.ceph.com]: reset os type and version on server
2021-04-23T17:42:31.871 INFO:teuthology.lock.ops:Updating gibba022.front.sepia.ceph.com on lock server
2021-04-23T17:42:31.885 INFO:teuthology.lock.ops:Node 'ubuntu@gibba022.front.sepia.ceph.com' reimaging is complete
2021-04-23T17:42:32.065 INFO:teuthology.provision.fog.gibba015:Scheduling deploy of rhel 8.3
2021-04-23T17:42:32.074 INFO:teuthology.provision.fog.gibba022:Scheduling deploy of rhel 8.3
2021-04-23T17:42:32.365 INFO:teuthology.orchestra.console:Power off gibba015
2021-04-23T17:42:32.383 INFO:teuthology.orchestra.console:Power off gibba022
2021-04-23T17:42:40.773 INFO:teuthology.orchestra.console:Power off for gibba015 completed
2021-04-23T17:42:40.778 INFO:teuthology.orchestra.console:Power off for gibba022 completed
2021-04-23T17:42:40.874 INFO:teuthology.orchestra.console:Power on gibba015
2021-04-23T17:42:40.881 INFO:teuthology.orchestra.console:Power on gibba022
2021-04-23T17:42:49.245 INFO:teuthology.orchestra.console:Power on for gibba015 completed
2021-04-23T17:42:49.251 INFO:teuthology.orchestra.console:Power on for gibba022 completed
2021-04-23T17:42:49.346 INFO:teuthology.provision.fog.gibba015:Waiting for deploy to finish
2021-04-23T17:42:49.351 INFO:teuthology.provision.fog.gibba022:Waiting for deploy to finish
2021-04-23T17:47:15.151 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-04-23T17:47:15.153 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-04-23T17:47:15.425 INFO:teuthology.provision.fog.gibba015:Node is ready
2021-04-23T17:47:15.435 INFO:teuthology.orchestra.run.gibba015.stdout:gibba015.front.sepia.ceph.com
2021-04-23T17:47:15.439 INFO:teuthology.provision.fog.gibba022:Node is ready
2021-04-23T17:47:15.450 INFO:teuthology.orchestra.run.gibba022.stdout:gibba022.front.sepia.ceph.com
2021-04-23T17:47:15.488 INFO:teuthology.orchestra.run.gibba015.stdout:172.21.2.115 gibba015.front.sepia.ceph.com gibba015
2021-04-23T17:47:15.502 INFO:teuthology.orchestra.run.gibba022.stdout:172.21.2.122 gibba022.front.sepia.ceph.com gibba022
2021-04-23T17:47:15.782 INFO:teuthology.provision.fog.gibba015:Deploy complete!
2021-04-23T17:47:15.789 INFO:teuthology.provision.fog.gibba022:Deploy complete!
2021-04-23T17:47:15.912 INFO:teuthology.lock.ops:Checking gibba015.front.sepia.ceph.com
2021-04-23T17:47:15.912 INFO:teuthology.lock.ops:New key found. Updating...
2021-04-23T17:47:15.961 INFO:teuthology.lock.ops:Checking gibba022.front.sepia.ceph.com
2021-04-23T17:47:15.961 INFO:teuthology.lock.ops:New key found. Updating...
2021-04-23T17:47:15.977 INFO:teuthology.lock.ops:Updating [gibba015.front.sepia.ceph.com]: set os type and version on server
2021-04-23T17:47:15.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-23T17:47:16.218 INFO:teuthology.orchestra.run.gibba015.stdout:x86_64
2021-04-23T17:47:16.270 INFO:teuthology.orchestra.run.gibba015.stdout:NAME="Red Hat Enterprise Linux"
2021-04-23T17:47:16.271 INFO:teuthology.orchestra.run.gibba015.stdout:VERSION="8.3 (Ootpa)"
2021-04-23T17:47:16.271 INFO:teuthology.orchestra.run.gibba015.stdout:ID="rhel"
2021-04-23T17:47:16.271 INFO:teuthology.orchestra.run.gibba015.stdout:ID_LIKE="fedora"
2021-04-23T17:47:16.271 INFO:teuthology.orchestra.run.gibba015.stdout:VERSION_ID="8.3"
2021-04-23T17:47:16.271 INFO:teuthology.orchestra.run.gibba015.stdout:PLATFORM_ID="platform:el8"
2021-04-23T17:47:16.271 INFO:teuthology.orchestra.run.gibba015.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
2021-04-23T17:47:16.271 INFO:teuthology.orchestra.run.gibba015.stdout:ANSI_COLOR="0;31"
2021-04-23T17:47:16.271 INFO:teuthology.orchestra.run.gibba015.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
2021-04-23T17:47:16.271 INFO:teuthology.orchestra.run.gibba015.stdout:HOME_URL="https://www.redhat.com/"
2021-04-23T17:47:16.271 INFO:teuthology.orchestra.run.gibba015.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-04-23T17:47:16.272 INFO:teuthology.orchestra.run.gibba015.stdout:
2021-04-23T17:47:16.272 INFO:teuthology.orchestra.run.gibba015.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2021-04-23T17:47:16.272 INFO:teuthology.orchestra.run.gibba015.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
2021-04-23T17:47:16.272 INFO:teuthology.orchestra.run.gibba015.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2021-04-23T17:47:16.272 INFO:teuthology.orchestra.run.gibba015.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3"
2021-04-23T17:47:16.273 INFO:teuthology.lock.ops:Updating gibba015.front.sepia.ceph.com on lock server
2021-04-23T17:47:16.291 INFO:teuthology.lock.ops:Updating [gibba022.front.sepia.ceph.com]: set os type and version on server
2021-04-23T17:47:16.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-23T17:47:16.535 INFO:teuthology.orchestra.run.gibba022.stdout:x86_64
2021-04-23T17:47:16.587 INFO:teuthology.orchestra.run.gibba022.stdout:NAME="Red Hat Enterprise Linux"
2021-04-23T17:47:16.588 INFO:teuthology.orchestra.run.gibba022.stdout:VERSION="8.3 (Ootpa)"
2021-04-23T17:47:16.588 INFO:teuthology.orchestra.run.gibba022.stdout:ID="rhel"
2021-04-23T17:47:16.588 INFO:teuthology.orchestra.run.gibba022.stdout:ID_LIKE="fedora"
2021-04-23T17:47:16.588 INFO:teuthology.orchestra.run.gibba022.stdout:VERSION_ID="8.3"
2021-04-23T17:47:16.588 INFO:teuthology.orchestra.run.gibba022.stdout:PLATFORM_ID="platform:el8"
2021-04-23T17:47:16.588 INFO:teuthology.orchestra.run.gibba022.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
2021-04-23T17:47:16.588 INFO:teuthology.orchestra.run.gibba022.stdout:ANSI_COLOR="0;31"
2021-04-23T17:47:16.588 INFO:teuthology.orchestra.run.gibba022.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
2021-04-23T17:47:16.588 INFO:teuthology.orchestra.run.gibba022.stdout:HOME_URL="https://www.redhat.com/"
2021-04-23T17:47:16.588 INFO:teuthology.orchestra.run.gibba022.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-04-23T17:47:16.589 INFO:teuthology.orchestra.run.gibba022.stdout:
2021-04-23T17:47:16.589 INFO:teuthology.orchestra.run.gibba022.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2021-04-23T17:47:16.589 INFO:teuthology.orchestra.run.gibba022.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
2021-04-23T17:47:16.589 INFO:teuthology.orchestra.run.gibba022.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2021-04-23T17:47:16.589 INFO:teuthology.orchestra.run.gibba022.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3"
2021-04-23T17:47:16.589 INFO:teuthology.lock.ops:Updating gibba022.front.sepia.ceph.com on lock server
2021-04-23T17:47:16.669 INFO:teuthology.dispatcher.supervisor:Running job 6067887
2021-04-23T17:47:16.669 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-04-23_08:19:18-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6067887 --name rfriedma-2021-04-23_08:19:18-rados-wip-ronenf-scrub-sched-distro-basic-gibba --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} -- /home/teuthworker/archive/rfriedma-2021-04-23_08:19:18-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6067887/orig.config.yaml
2021-04-23T17:47:16.672 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-04-23_08:19:18-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6067887
2021-04-23T17:47:16.672 INFO:teuthology.dispatcher.supervisor:Job PID: 31733
2021-04-23T17:47:16.672 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-04-24T05:46:42.423 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/rfriedma-2021-04-23_08:19:18-rados-wip-ronenf-scrub-sched-distro-basic-gibba/jobs/ failed with status 504:
504 Gateway Time-out
504 Gateway Time-out
nginx/1.10.3 (Ubuntu)
2021-04-24T05:49:08.473 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-04-24T05:49:08.644 INFO:teuthology.kill:Killing Pids: {31733}
2021-04-24T05:49:27.581 INFO:teuthology.kill:Nuking machines: ['gibba015', 'gibba022']
2021-04-24T05:49:28.572 INFO:teuthology.kill:2021-04-24 05:49:28,572.572 INFO:teuthology.nuke:targets:
2021-04-24T05:49:28.573 INFO:teuthology.kill: gibba015.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+1w6wLc6IWkviCp9xCpChQs3Tx3PVTWJtOoeuUi9oTbCNfTLT6FrI2Yx4bY0Tp5HWrvsSMGkIx+qf41COeAOcyrvvNiOKyMqra0CW4SLnLQFqVZYRaxHJfRfWLTg7KhfYS4NEPEGBL6NhiYgpACsXontz34dUPOwgRyRo4YCERHdl2IhME1zRYkWgPKpvo4UpFYEC4SywZ9ZZiTVvJK+1d8IO5rs2s3kuAgxBrDrganta31eOneBNM+kT9Rc4sfvk2AWDwfQEFAN+3wwBA12eZoEro/6LGSWobU7jQx3Xv3d9M9o2W4aJn9yDnIhT/6E6VJNu3F0WiYLIs03tG1n8kSam0rbnl6VAy2bWD5puJVMvzKdTgLuMkC/JVvz3RBWMboMjXMlJ3mbvTr0jgFQimojewUEcaaAVivfr1M8tB/b57AGltyEHaWDx+DeZi9EV2re/tOZr4rLFgsNTGwVMF7CCCbaIjtWcdFkOeu9VDuuAsIzSsswcyKJX9ggYa5c=
2021-04-24T05:49:28.573 INFO:teuthology.kill: gibba022.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD4xarJqIXHJE36ojREBcpW9hNK91aKA3gaOOt2nfYiw72xX48/28XDeKRE1sMwg0JbVmFOJ4TBzxzi3rA2vFbQGhM+B6wyzF/EPGIe5/EN+vpn9seo7Y8ws+w0WpUMb6i0bLSlQ373ZTRppHdQK1z3UiSXN6OyTvQjIpgwimqfx1TREB6iM+q4Da5GV5TvwUrtrlcDtvrDmHwk/ikQe0s2PX07xX8SrZ009qQsq6JB2ggi4pac1wh+iY8+1U1sEKaBLu4s1nkpM96I16KlMF1m1L1aivAP6+HUINykRslY6QSI8SkT1+JkiUuEiQk1vDBWpWn45iACvcnE0GB4YupK5ebX2ZEEqPdOjLu+6kpT82FyFWEdDDzZvPgtVOfLU+1nPMJjY6QhpQ3paF3FES6s5BmTQhsWNwO+nDuOVU7D7JjlRIcoAqAvkamyoJXJtXOHfPolOAyGZSgoGVuAgu+INqsA+lWnp19fecVnnujx+/r2D83Z9B7kCnp0nGn1FLU=
2021-04-24T05:49:28.573 INFO:teuthology.kill:2021-04-24 05:49:28,573.573 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-24T05:49:28.575 INFO:teuthology.kill:2021-04-24 05:49:28,575.575 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-24T05:49:29.896 INFO:teuthology.kill:2021-04-24 05:49:29,896.896 INFO:teuthology.orchestra.console:Power off gibba015
2021-04-24T05:49:29.912 INFO:teuthology.kill:2021-04-24 05:49:29,912.912 INFO:teuthology.orchestra.console:Power off gibba022
2021-04-24T05:49:38.302 INFO:teuthology.kill:2021-04-24 05:49:38,302.302 INFO:teuthology.orchestra.console:Power off for gibba015 completed
2021-04-24T05:49:38.308 INFO:teuthology.kill:2021-04-24 05:49:38,308.308 INFO:teuthology.orchestra.console:Power off for gibba022 completed
2021-04-24T05:49:38.576 INFO:teuthology.task.internal:roles: ubuntu@gibba015.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2021-04-24T05:49:38.577 INFO:teuthology.task.internal:roles: ubuntu@gibba022.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2021-04-24T05:49:38.577 INFO:teuthology.misc:Compressing logs...
2021-04-24T05:49:38.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-24T05:49:56.956 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
teuth_config.archive_base, job_config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
compress_logs(ctx, log_path)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/misc.py", line 1378, in compress_logs
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/remote.py", line 215, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 434, in run
transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-04-24T05:49:57.012 INFO:teuthology.kill:No teuthology processes running
2021-04-24T05:49:58.247 INFO:teuthology.kill:Nuking machines: ['gibba015', 'gibba022']
2021-04-24T05:49:59.168 INFO:teuthology.kill:2021-04-24 05:49:59,168.168 INFO:teuthology.nuke:targets:
2021-04-24T05:49:59.169 INFO:teuthology.kill: gibba015.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+1w6wLc6IWkviCp9xCpChQs3Tx3PVTWJtOoeuUi9oTbCNfTLT6FrI2Yx4bY0Tp5HWrvsSMGkIx+qf41COeAOcyrvvNiOKyMqra0CW4SLnLQFqVZYRaxHJfRfWLTg7KhfYS4NEPEGBL6NhiYgpACsXontz34dUPOwgRyRo4YCERHdl2IhME1zRYkWgPKpvo4UpFYEC4SywZ9ZZiTVvJK+1d8IO5rs2s3kuAgxBrDrganta31eOneBNM+kT9Rc4sfvk2AWDwfQEFAN+3wwBA12eZoEro/6LGSWobU7jQx3Xv3d9M9o2W4aJn9yDnIhT/6E6VJNu3F0WiYLIs03tG1n8kSam0rbnl6VAy2bWD5puJVMvzKdTgLuMkC/JVvz3RBWMboMjXMlJ3mbvTr0jgFQimojewUEcaaAVivfr1M8tB/b57AGltyEHaWDx+DeZi9EV2re/tOZr4rLFgsNTGwVMF7CCCbaIjtWcdFkOeu9VDuuAsIzSsswcyKJX9ggYa5c=
2021-04-24T05:49:59.169 INFO:teuthology.kill: gibba022.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD4xarJqIXHJE36ojREBcpW9hNK91aKA3gaOOt2nfYiw72xX48/28XDeKRE1sMwg0JbVmFOJ4TBzxzi3rA2vFbQGhM+B6wyzF/EPGIe5/EN+vpn9seo7Y8ws+w0WpUMb6i0bLSlQ373ZTRppHdQK1z3UiSXN6OyTvQjIpgwimqfx1TREB6iM+q4Da5GV5TvwUrtrlcDtvrDmHwk/ikQe0s2PX07xX8SrZ009qQsq6JB2ggi4pac1wh+iY8+1U1sEKaBLu4s1nkpM96I16KlMF1m1L1aivAP6+HUINykRslY6QSI8SkT1+JkiUuEiQk1vDBWpWn45iACvcnE0GB4YupK5ebX2ZEEqPdOjLu+6kpT82FyFWEdDDzZvPgtVOfLU+1nPMJjY6QhpQ3paF3FES6s5BmTQhsWNwO+nDuOVU7D7JjlRIcoAqAvkamyoJXJtXOHfPolOAyGZSgoGVuAgu+INqsA+lWnp19fecVnnujx+/r2D83Z9B7kCnp0nGn1FLU=
2021-04-24T05:50:40.899 INFO:teuthology.kill:2021-04-24 05:50:40,898.898 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-24T05:50:40.909 INFO:teuthology.kill:2021-04-24 05:50:40,909.909 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-24T05:50:43.586 INFO:teuthology.kill:2021-04-24 05:50:43,586.586 INFO:teuthology.orchestra.console:Power off gibba015
2021-04-24T05:50:43.603 INFO:teuthology.kill:2021-04-24 05:50:43,603.603 INFO:teuthology.orchestra.console:Power off gibba022
2021-04-24T05:50:43.743 INFO:teuthology.kill:2021-04-24 05:50:43,743.743 INFO:teuthology.orchestra.console:Power off for gibba015 completed
2021-04-24T05:50:43.751 INFO:teuthology.kill:2021-04-24 05:50:43,751.751 INFO:teuthology.orchestra.console:Power off for gibba022 completed
2021-04-24T05:50:43.893 INFO:teuthology.kill:2021-04-24 05:50:43,893.893 INFO:teuthology.lock.ops:unlocked gibba015.front.sepia.ceph.com
2021-04-24T05:50:43.910 INFO:teuthology.kill:2021-04-24 05:50:43,910.910 INFO:teuthology.lock.ops:unlocked gibba022.front.sepia.ceph.com
2021-04-24T05:52:44.209 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-04-24T05:52:44.335 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-04-24T05:52:44.336 INFO:teuthology.nuke:Checking targets against current locks
2021-04-24T05:52:44.409 INFO:teuthology.nuke:Not nuking gibba015.front.sepia.ceph.com because description doesn't match
2021-04-24T05:52:44.410 INFO:teuthology.nuke:Not nuking gibba022.front.sepia.ceph.com because description doesn't match