2021-02-18T19:56:09.718 INFO:root:teuthology version: 1.1.0-e337964c 2021-02-18T19:56:09.850 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba018.front.sepia.ceph.com' reimaging 2021-02-18T19:56:09.851 INFO:teuthology.lock.ops:Updating [ubuntu@gibba018.front.sepia.ceph.com]: reset os type and version on server 2021-02-18T19:56:09.851 INFO:teuthology.lock.ops:Updating gibba018.front.sepia.ceph.com on lock server 2021-02-18T19:56:09.869 INFO:teuthology.lock.ops:Node 'ubuntu@gibba018.front.sepia.ceph.com' reimaging is complete 2021-02-18T19:56:09.869 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba027.front.sepia.ceph.com' reimaging 2021-02-18T19:56:09.869 INFO:teuthology.lock.ops:Updating [ubuntu@gibba027.front.sepia.ceph.com]: reset os type and version on server 2021-02-18T19:56:09.869 INFO:teuthology.lock.ops:Updating gibba027.front.sepia.ceph.com on lock server 2021-02-18T19:56:09.888 INFO:teuthology.lock.ops:Node 'ubuntu@gibba027.front.sepia.ceph.com' reimaging is complete 2021-02-18T19:56:10.073 INFO:teuthology.provision.fog.gibba018:Scheduling deploy of ubuntu 18.04 2021-02-18T19:56:10.089 INFO:teuthology.provision.fog.gibba027:Scheduling deploy of ubuntu 18.04 2021-02-18T19:56:10.398 INFO:teuthology.orchestra.console:Power off gibba027 2021-02-18T19:56:10.411 INFO:teuthology.orchestra.console:Power off gibba018 2021-02-18T19:56:22.901 INFO:teuthology.orchestra.console:Power off for gibba027 completed 2021-02-18T19:56:22.920 INFO:teuthology.orchestra.console:Power off for gibba018 completed 2021-02-18T19:56:23.003 INFO:teuthology.orchestra.console:Power on gibba027 2021-02-18T19:56:23.022 INFO:teuthology.orchestra.console:Power on gibba018 2021-02-18T19:56:31.388 INFO:teuthology.orchestra.console:Power on for gibba027 completed 2021-02-18T19:56:31.395 INFO:teuthology.orchestra.console:Power on for gibba018 completed 2021-02-18T19:56:31.489 INFO:teuthology.provision.fog.gibba027:Waiting for deploy to finish 2021-02-18T19:56:31.497 INFO:teuthology.provision.fog.gibba018:Waiting for deploy to finish 2021-02-18T20:00:58.560 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-18T20:01:04.676 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-18T20:01:10.688 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-18T20:01:16.777 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-18T20:01:17.238 INFO:teuthology.provision.fog.gibba027:Node is ready 2021-02-18T20:01:17.289 INFO:teuthology.orchestra.run.gibba027.stdout:gibba027.front.sepia.ceph.com 2021-02-18T20:01:17.339 INFO:teuthology.orchestra.run.gibba027.stdout:172.21.2.127 gibba027.front.sepia.ceph.com gibba027 2021-02-18T20:01:17.677 INFO:teuthology.provision.fog.gibba027:Deploy complete! 2021-02-18T20:01:21.147 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-18T20:01:21.629 INFO:teuthology.provision.fog.gibba018:Node is ready 2021-02-18T20:01:21.677 INFO:teuthology.orchestra.run.gibba018.stdout:gibba018.front.sepia.ceph.com 2021-02-18T20:01:21.727 INFO:teuthology.orchestra.run.gibba018.stdout:172.21.2.118 gibba018.front.sepia.ceph.com gibba018 2021-02-18T20:01:22.035 INFO:teuthology.provision.fog.gibba018:Deploy complete! 2021-02-18T20:01:22.166 INFO:teuthology.lock.ops:Checking gibba018.front.sepia.ceph.com 2021-02-18T20:01:22.166 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-18T20:01:22.182 INFO:teuthology.lock.ops:Checking gibba027.front.sepia.ceph.com 2021-02-18T20:01:22.182 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-18T20:01:22.197 INFO:teuthology.lock.ops:Updating [gibba018.front.sepia.ceph.com]: set os type and version on server 2021-02-18T20:01:22.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-18T20:01:22.464 INFO:teuthology.orchestra.run.gibba018.stdout:x86_64 2021-02-18T20:01:22.517 INFO:teuthology.orchestra.run.gibba018.stdout:NAME="Ubuntu" 2021-02-18T20:01:22.517 INFO:teuthology.orchestra.run.gibba018.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-18T20:01:22.517 INFO:teuthology.orchestra.run.gibba018.stdout:ID=ubuntu 2021-02-18T20:01:22.517 INFO:teuthology.orchestra.run.gibba018.stdout:ID_LIKE=debian 2021-02-18T20:01:22.517 INFO:teuthology.orchestra.run.gibba018.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-18T20:01:22.517 INFO:teuthology.orchestra.run.gibba018.stdout:VERSION_ID="18.04" 2021-02-18T20:01:22.517 INFO:teuthology.orchestra.run.gibba018.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-18T20:01:22.517 INFO:teuthology.orchestra.run.gibba018.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-18T20:01:22.517 INFO:teuthology.orchestra.run.gibba018.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-18T20:01:22.518 INFO:teuthology.orchestra.run.gibba018.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-18T20:01:22.518 INFO:teuthology.orchestra.run.gibba018.stdout:VERSION_CODENAME=bionic 2021-02-18T20:01:22.518 INFO:teuthology.orchestra.run.gibba018.stdout:UBUNTU_CODENAME=bionic 2021-02-18T20:01:22.519 INFO:teuthology.lock.ops:Updating gibba018.front.sepia.ceph.com on lock server 2021-02-18T20:01:22.534 INFO:teuthology.lock.ops:Updating [gibba027.front.sepia.ceph.com]: set os type and version on server 2021-02-18T20:01:22.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-18T20:01:22.833 INFO:teuthology.orchestra.run.gibba027.stdout:x86_64 2021-02-18T20:01:22.883 INFO:teuthology.orchestra.run.gibba027.stdout:NAME="Ubuntu" 2021-02-18T20:01:22.883 INFO:teuthology.orchestra.run.gibba027.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-18T20:01:22.884 INFO:teuthology.orchestra.run.gibba027.stdout:ID=ubuntu 2021-02-18T20:01:22.884 INFO:teuthology.orchestra.run.gibba027.stdout:ID_LIKE=debian 2021-02-18T20:01:22.884 INFO:teuthology.orchestra.run.gibba027.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-18T20:01:22.884 INFO:teuthology.orchestra.run.gibba027.stdout:VERSION_ID="18.04" 2021-02-18T20:01:22.884 INFO:teuthology.orchestra.run.gibba027.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-18T20:01:22.884 INFO:teuthology.orchestra.run.gibba027.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-18T20:01:22.884 INFO:teuthology.orchestra.run.gibba027.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-18T20:01:22.884 INFO:teuthology.orchestra.run.gibba027.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-18T20:01:22.884 INFO:teuthology.orchestra.run.gibba027.stdout:VERSION_CODENAME=bionic 2021-02-18T20:01:22.884 INFO:teuthology.orchestra.run.gibba027.stdout:UBUNTU_CODENAME=bionic 2021-02-18T20:01:22.885 INFO:teuthology.lock.ops:Updating gibba027.front.sepia.ceph.com on lock server 2021-02-18T20:01:22.970 INFO:teuthology.dispatcher.supervisor:Running job 5892906 2021-02-18T20:01:22.971 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_e337964ce0a8ac73743e4eb5665a736ad9f9a048/virtualenv/bin/teuthology -v --owner scheduled_ideepika@teuthology --archive /home/teuthworker/archive/ideepika-2021-02-18_19:47:00-rados:thrash-erasure-code-test-wip-tracker-48609-distro-basic-gibba/5892906 --name ideepika-2021-02-18_19:47:00-rados:thrash-erasure-code-test-wip-tracker-48609-distro-basic-gibba --description rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} -- /home/teuthworker/archive/ideepika-2021-02-18_19:47:00-rados:thrash-erasure-code-test-wip-tracker-48609-distro-basic-gibba/5892906/orig.config.yaml 2021-02-18T20:01:22.979 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/ideepika-2021-02-18_19:47:00-rados:thrash-erasure-code-test-wip-tracker-48609-distro-basic-gibba/5892906 2021-02-18T20:01:22.979 INFO:teuthology.dispatcher.supervisor:Job PID: 30711 2021-02-18T20:01:22.979 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-02-18T20:02:55.384 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-19T08:01:39.254 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-02-19T08:01:40.090 INFO:teuthology.kill:Killing Pids: {30711} 2021-02-19T08:01:42.185 INFO:teuthology.kill:Nuking machines: ['gibba018', 'gibba027'] 2021-02-19T08:01:44.070 INFO:teuthology.kill:2021-02-19 08:01:44,070.070 INFO:teuthology.nuke:targets: 2021-02-19T08:01:44.070 INFO:teuthology.kill: gibba018.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPpeD7wIDzEOwUi1nC0KhsxSzbuIRUP4EsbrCE1iOWASE/l6rLZzbr/WHjeo2alQxeSoWxCy/ZsHXYim8337pTKpeA5VS/hqvUfMkmQOy03VqzwGIbzdTXaKQ2x57Kzf0R2ycrKt4xEfLi/Wvd963yb/NJ/pLw8fzi1lX5BAttLzCbuaLm6pasCiQ9bW78Av3v5m4gIXYQcTjthjEwffZYyg5ayF2d+o8e69FwHeHLJTU8NwSBo/Vk7PkX47bQjTuPobJ5rdttsHEeLP7+kc5kxb4SljFM1g33bl21ND7RR1txaNJ/N79JVSY6HgD2UPlk6gb9+MztKWCanEq5VZjV 2021-02-19T08:01:44.071 INFO:teuthology.kill: gibba027.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDw0JEfRISP8caqulxXiPe9katoKmTrXirqq3vcyDX3Sp110Se3rcd9DD7BcO0MCMoiaNjI+t1sIdQ5OOzqikkmU0jbIDl1Jy+3frGp7eoLSB24rQq/A90d978GA4q41kbHL3LInhQZ5OJhPs6hMEcZF2od0w143IH93bcbvYYBgrMAmUlxZ2EJ4UaG/sqoCJz1BT1B+eDWlQZzHDdlvZ9HBgd4rfkAQEcICHVkRjoBznnk/9guD14p1fs1S/aVJTbyx+t2gpTZ5SoeZit0kJgbNkutEOY//nSFTo8WT50TwYhnewIC4jgawtwVzCVnMOgQBsE0T1A59lkiZS5RVl+p 2021-02-19T08:01:44.071 INFO:teuthology.kill:2021-02-19 08:01:44,070.070 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-19T08:01:44.085 INFO:teuthology.kill:2021-02-19 08:01:44,085.085 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-19T08:01:44.165 INFO:teuthology.kill:2021-02-19 08:01:44,164.164 INFO:teuthology.orchestra.console:Power off gibba027 2021-02-19T08:01:44.185 INFO:teuthology.kill:2021-02-19 08:01:44,182.182 INFO:teuthology.orchestra.console:Power off gibba018 2021-02-19T08:01:56.704 INFO:teuthology.kill:2021-02-19 08:01:56,704.704 INFO:teuthology.orchestra.console:Power off for gibba027 completed 2021-02-19T08:01:56.711 INFO:teuthology.kill:2021-02-19 08:01:56,710.710 INFO:teuthology.orchestra.console:Power off for gibba018 completed 2021-02-19T08:01:56.998 INFO:teuthology.task.internal:roles: ubuntu@gibba018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-02-19T08:01:56.998 INFO:teuthology.task.internal:roles: ubuntu@gibba027.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-02-19T08:01:56.999 INFO:teuthology.misc:Compressing logs... 2021-02-19T08:01:56.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-19T08:02:30.691 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_e337964ce0a8ac73743e4eb5665a736ad9f9a048/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e337964ce0a8ac73743e4eb5665a736ad9f9a048/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e337964ce0a8ac73743e4eb5665a736ad9f9a048/teuthology/misc.py", line 1372, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_e337964ce0a8ac73743e4eb5665a736ad9f9a048/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_e337964ce0a8ac73743e4eb5665a736ad9f9a048/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_e337964ce0a8ac73743e4eb5665a736ad9f9a048/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e337964ce0a8ac73743e4eb5665a736ad9f9a048/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-02-19T08:02:30.818 INFO:teuthology.kill:No teuthology processes running 2021-02-19T08:02:33.209 INFO:teuthology.kill:Nuking machines: ['gibba018', 'gibba027'] 2021-02-19T08:02:35.814 INFO:teuthology.kill:2021-02-19 08:02:35,813.813 INFO:teuthology.nuke:targets: 2021-02-19T08:02:35.814 INFO:teuthology.kill: gibba018.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPpeD7wIDzEOwUi1nC0KhsxSzbuIRUP4EsbrCE1iOWASE/l6rLZzbr/WHjeo2alQxeSoWxCy/ZsHXYim8337pTKpeA5VS/hqvUfMkmQOy03VqzwGIbzdTXaKQ2x57Kzf0R2ycrKt4xEfLi/Wvd963yb/NJ/pLw8fzi1lX5BAttLzCbuaLm6pasCiQ9bW78Av3v5m4gIXYQcTjthjEwffZYyg5ayF2d+o8e69FwHeHLJTU8NwSBo/Vk7PkX47bQjTuPobJ5rdttsHEeLP7+kc5kxb4SljFM1g33bl21ND7RR1txaNJ/N79JVSY6HgD2UPlk6gb9+MztKWCanEq5VZjV 2021-02-19T08:02:35.814 INFO:teuthology.kill: gibba027.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDw0JEfRISP8caqulxXiPe9katoKmTrXirqq3vcyDX3Sp110Se3rcd9DD7BcO0MCMoiaNjI+t1sIdQ5OOzqikkmU0jbIDl1Jy+3frGp7eoLSB24rQq/A90d978GA4q41kbHL3LInhQZ5OJhPs6hMEcZF2od0w143IH93bcbvYYBgrMAmUlxZ2EJ4UaG/sqoCJz1BT1B+eDWlQZzHDdlvZ9HBgd4rfkAQEcICHVkRjoBznnk/9guD14p1fs1S/aVJTbyx+t2gpTZ5SoeZit0kJgbNkutEOY//nSFTo8WT50TwYhnewIC4jgawtwVzCVnMOgQBsE0T1A59lkiZS5RVl+p 2021-02-19T08:02:35.846 INFO:teuthology.kill:2021-02-19 08:02:35,846.846 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-19T08:02:35.856 INFO:teuthology.kill:2021-02-19 08:02:35,856.856 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-19T08:02:35.926 INFO:teuthology.kill:2021-02-19 08:02:35,925.925 INFO:teuthology.orchestra.console:Power off gibba018 2021-02-19T08:02:35.942 INFO:teuthology.kill:2021-02-19 08:02:35,942.942 INFO:teuthology.orchestra.console:Power off gibba027 2021-02-19T08:02:36.080 INFO:teuthology.kill:2021-02-19 08:02:36,079.079 INFO:teuthology.orchestra.console:Power off for gibba018 completed 2021-02-19T08:02:36.095 INFO:teuthology.kill:2021-02-19 08:02:36,095.095 INFO:teuthology.orchestra.console:Power off for gibba027 completed 2021-02-19T08:02:36.230 INFO:teuthology.kill:2021-02-19 08:02:36,230.230 INFO:teuthology.lock.ops:unlocked gibba018.front.sepia.ceph.com 2021-02-19T08:02:36.237 INFO:teuthology.kill:2021-02-19 08:02:36,236.236 INFO:teuthology.lock.ops:unlocked gibba027.front.sepia.ceph.com 2021-02-19T08:04:36.441 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-02-19T08:04:36.557 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-02-19T08:04:36.557 INFO:teuthology.nuke:Checking targets against current locks 2021-02-19T08:04:36.656 INFO:teuthology.nuke:Not nuking gibba018.front.sepia.ceph.com because description doesn't match 2021-02-19T08:04:36.657 INFO:teuthology.nuke:Not nuking gibba027.front.sepia.ceph.com because description doesn't match