2021-02-14T18:43:08.618 INFO:root:teuthology version: 1.1.0-64d5f180 2021-02-14T18:43:09.029 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi031.front.sepia.ceph.com' reimaging 2021-02-14T18:43:09.030 INFO:teuthology.lock.ops:Updating [ubuntu@smithi031.front.sepia.ceph.com]: reset os type and version on server 2021-02-14T18:43:09.030 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2021-02-14T18:43:09.056 INFO:teuthology.lock.ops:Node 'ubuntu@smithi031.front.sepia.ceph.com' reimaging is complete 2021-02-14T18:43:09.056 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi080.front.sepia.ceph.com' reimaging 2021-02-14T18:43:09.057 INFO:teuthology.lock.ops:Updating [ubuntu@smithi080.front.sepia.ceph.com]: reset os type and version on server 2021-02-14T18:43:09.057 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server 2021-02-14T18:43:09.110 INFO:teuthology.lock.ops:Node 'ubuntu@smithi080.front.sepia.ceph.com' reimaging is complete 2021-02-14T18:43:09.437 INFO:teuthology.provision.fog.smithi031:Scheduling deploy of ubuntu 18.04 2021-02-14T18:43:09.457 INFO:teuthology.provision.fog.smithi080:Scheduling deploy of ubuntu 18.04 2021-02-14T18:43:09.925 INFO:teuthology.orchestra.console:Power off smithi031 2021-02-14T18:43:10.031 INFO:teuthology.orchestra.console:Power off smithi080 2021-02-14T18:43:18.491 INFO:teuthology.orchestra.console:Power off for smithi031 completed 2021-02-14T18:43:18.592 INFO:teuthology.orchestra.console:Power on smithi031 2021-02-14T18:43:22.647 INFO:teuthology.orchestra.console:Power off for smithi080 completed 2021-02-14T18:43:22.749 INFO:teuthology.orchestra.console:Power on smithi080 2021-02-14T18:43:27.061 INFO:teuthology.orchestra.console:Power on for smithi031 completed 2021-02-14T18:43:27.080 INFO:teuthology.orchestra.console:Power on for smithi080 completed 2021-02-14T18:43:27.162 INFO:teuthology.provision.fog.smithi031:Waiting for deploy to finish 2021-02-14T18:43:27.181 INFO:teuthology.provision.fog.smithi080:Waiting for deploy to finish 2021-02-14T18:47:28.582 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-14T18:47:34.615 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-14T18:47:40.629 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-14T18:47:46.641 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-14T18:47:52.733 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-14T18:47:53.188 INFO:teuthology.provision.fog.smithi031:Node is ready 2021-02-14T18:47:53.233 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031.front.sepia.ceph.com 2021-02-14T18:47:53.282 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31 smithi031.front.sepia.ceph.com smithi031 2021-02-14T18:47:53.581 INFO:teuthology.provision.fog.smithi031:Deploy complete! 2021-02-14T18:48:30.204 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-14T18:48:30.620 INFO:teuthology.provision.fog.smithi080:Node is ready 2021-02-14T18:48:30.667 INFO:teuthology.orchestra.run.smithi080.stdout:smithi080.front.sepia.ceph.com 2021-02-14T18:48:30.716 INFO:teuthology.orchestra.run.smithi080.stdout:172.21.15.80 smithi080.front.sepia.ceph.com smithi080 2021-02-14T18:48:31.038 INFO:teuthology.provision.fog.smithi080:Deploy complete! 2021-02-14T18:48:31.190 INFO:teuthology.lock.ops:Checking smithi031.front.sepia.ceph.com 2021-02-14T18:48:31.190 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-14T18:48:31.207 INFO:teuthology.lock.ops:Checking smithi080.front.sepia.ceph.com 2021-02-14T18:48:31.207 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-14T18:48:31.223 INFO:teuthology.lock.ops:Updating [smithi031.front.sepia.ceph.com]: set os type and version on server 2021-02-14T18:48:31.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-14T18:48:31.573 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2021-02-14T18:48:31.621 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="Ubuntu" 2021-02-14T18:48:31.621 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-14T18:48:31.621 INFO:teuthology.orchestra.run.smithi031.stdout:ID=ubuntu 2021-02-14T18:48:31.621 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE=debian 2021-02-14T18:48:31.621 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-14T18:48:31.621 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="18.04" 2021-02-14T18:48:31.621 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-14T18:48:31.621 INFO:teuthology.orchestra.run.smithi031.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-14T18:48:31.621 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-14T18:48:31.622 INFO:teuthology.orchestra.run.smithi031.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-14T18:48:31.622 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_CODENAME=bionic 2021-02-14T18:48:31.622 INFO:teuthology.orchestra.run.smithi031.stdout:UBUNTU_CODENAME=bionic 2021-02-14T18:48:31.623 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2021-02-14T18:48:31.638 INFO:teuthology.lock.ops:Updating [smithi080.front.sepia.ceph.com]: set os type and version on server 2021-02-14T18:48:31.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-14T18:48:31.999 INFO:teuthology.orchestra.run.smithi080.stdout:x86_64 2021-02-14T18:48:32.047 INFO:teuthology.orchestra.run.smithi080.stdout:NAME="Ubuntu" 2021-02-14T18:48:32.047 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-14T18:48:32.047 INFO:teuthology.orchestra.run.smithi080.stdout:ID=ubuntu 2021-02-14T18:48:32.047 INFO:teuthology.orchestra.run.smithi080.stdout:ID_LIKE=debian 2021-02-14T18:48:32.047 INFO:teuthology.orchestra.run.smithi080.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-14T18:48:32.047 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_ID="18.04" 2021-02-14T18:48:32.047 INFO:teuthology.orchestra.run.smithi080.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-14T18:48:32.048 INFO:teuthology.orchestra.run.smithi080.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-14T18:48:32.048 INFO:teuthology.orchestra.run.smithi080.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-14T18:48:32.048 INFO:teuthology.orchestra.run.smithi080.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-14T18:48:32.048 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_CODENAME=bionic 2021-02-14T18:48:32.048 INFO:teuthology.orchestra.run.smithi080.stdout:UBUNTU_CODENAME=bionic 2021-02-14T18:48:32.048 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server 2021-02-14T18:48:32.124 INFO:teuthology.dispatcher.supervisor:Running job 5881072 2021-02-14T18:48:32.124 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/virtualenv/bin/teuthology -v --owner scheduled_sseshasa@teuthology --archive /home/teuthworker/archive/sseshasa-2021-02-14_18:00:38-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5881072 --name sseshasa-2021-02-14_18:00:38-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects} -- /home/teuthworker/archive/sseshasa-2021-02-14_18:00:38-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5881072/orig.config.yaml 2021-02-14T18:48:32.128 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sseshasa-2021-02-14_18:00:38-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5881072 2021-02-14T18:48:32.128 INFO:teuthology.dispatcher.supervisor:Job PID: 15931 2021-02-14T18:48:32.129 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-02-15T06:48:49.156 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-02-15T06:48:49.449 INFO:teuthology.kill:Killing Pids: {15931} 2021-02-15T06:48:51.176 INFO:teuthology.kill:Nuking machines: ['smithi031', 'smithi080'] 2021-02-15T06:48:52.029 INFO:teuthology.kill:2021-02-15 06:48:52,028.028 INFO:teuthology.nuke:targets: 2021-02-15T06:48:52.029 INFO:teuthology.kill: smithi031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgrZcCShFREmSudPYOVmn48VpaUdKI/s28JQiojeY8nyv45gymXnO0aa6kEH8R+W9NZSh57eu6aEKbtr/ILwVh76vxqZnskZ9Q2DL/3GMM6eI7I8alL7GrKqWBufVnR4B36SDqM6RKGLZeQmzITod05Mhqr1DvJTRqMh2qa6Kn8l1R7T5jCOxoR26YIyZRZJWiM+CXcVVg89XfOH+sXlTpptSMsmneCEDm1sz09rRNk33oHsRznvcpD6YlDB05YcSZSkncuwgTc0QSrDw+dP3MxOd/M7NgIyhKdklW8zKBi8GUMLBfrWVXrdDyzfcDe1SbAFqbnbv1Bp9wzS/cJNh/ 2021-02-15T06:48:52.029 INFO:teuthology.kill: smithi080.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDG5LSZFYd7HqEIAbh025jouy/jIkn0gfBATeebN5nplnaWMkzm7L8rwxxhvOUDiqJ+VXdyV7u3poA0XtLcD8sWqytzZmkiPyU6RujOdUbbjpuJ72m5IGBr2bS0PpAP5NiUX3X7FyvFdMEeiPsUfDmzaXFdmMrNUuzf3yaDi/Yxnz8v7KOmsFJwdV/6Hwc3LYN6sGqL+itMPOIDcnl0r43kU9XcqJwdLCP0/Pyw8vScCgBrs3XJ7KG9vU+Bt8YDa6VE6msa5kkMvMiYVn/6P1nEjVLywRqktazzX+ap3GsCdXqeaMDjqPwfZIA+tzowjpvGfq1DR+Af68otevBsGynt 2021-02-15T06:48:52.029 INFO:teuthology.kill:2021-02-15 06:48:52,029.029 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-15T06:48:52.038 INFO:teuthology.kill:2021-02-15 06:48:52,038.038 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-15T06:48:52.109 INFO:teuthology.kill:2021-02-15 06:48:52,109.109 INFO:teuthology.orchestra.console:Power off smithi031 2021-02-15T06:48:52.135 INFO:teuthology.kill:2021-02-15 06:48:52,134.134 INFO:teuthology.orchestra.console:Power off smithi080 2021-02-15T06:49:00.516 INFO:teuthology.kill:2021-02-15 06:49:00,516.516 INFO:teuthology.orchestra.console:Power off for smithi031 completed 2021-02-15T06:49:04.634 INFO:teuthology.kill:2021-02-15 06:49:04,633.633 INFO:teuthology.orchestra.console:Power off for smithi080 completed 2021-02-15T06:49:04.869 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-02-15T06:49:04.869 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-02-15T06:49:04.869 INFO:teuthology.misc:Compressing logs... 2021-02-15T06:49:04.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-15T06:50:04.873 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/misc.py", line 1372, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-02-15T06:50:04.965 INFO:teuthology.kill:No teuthology processes running 2021-02-15T06:50:06.913 INFO:teuthology.kill:Nuking machines: ['smithi031', 'smithi080'] 2021-02-15T06:50:08.588 INFO:teuthology.kill:2021-02-15 06:50:08,588.588 INFO:teuthology.nuke:targets: 2021-02-15T06:50:08.589 INFO:teuthology.kill: smithi031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgrZcCShFREmSudPYOVmn48VpaUdKI/s28JQiojeY8nyv45gymXnO0aa6kEH8R+W9NZSh57eu6aEKbtr/ILwVh76vxqZnskZ9Q2DL/3GMM6eI7I8alL7GrKqWBufVnR4B36SDqM6RKGLZeQmzITod05Mhqr1DvJTRqMh2qa6Kn8l1R7T5jCOxoR26YIyZRZJWiM+CXcVVg89XfOH+sXlTpptSMsmneCEDm1sz09rRNk33oHsRznvcpD6YlDB05YcSZSkncuwgTc0QSrDw+dP3MxOd/M7NgIyhKdklW8zKBi8GUMLBfrWVXrdDyzfcDe1SbAFqbnbv1Bp9wzS/cJNh/ 2021-02-15T06:50:08.589 INFO:teuthology.kill: smithi080.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDG5LSZFYd7HqEIAbh025jouy/jIkn0gfBATeebN5nplnaWMkzm7L8rwxxhvOUDiqJ+VXdyV7u3poA0XtLcD8sWqytzZmkiPyU6RujOdUbbjpuJ72m5IGBr2bS0PpAP5NiUX3X7FyvFdMEeiPsUfDmzaXFdmMrNUuzf3yaDi/Yxnz8v7KOmsFJwdV/6Hwc3LYN6sGqL+itMPOIDcnl0r43kU9XcqJwdLCP0/Pyw8vScCgBrs3XJ7KG9vU+Bt8YDa6VE6msa5kkMvMiYVn/6P1nEjVLywRqktazzX+ap3GsCdXqeaMDjqPwfZIA+tzowjpvGfq1DR+Af68otevBsGynt 2021-02-15T06:50:08.614 INFO:teuthology.kill:2021-02-15 06:50:08,614.614 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-15T06:50:08.617 INFO:teuthology.kill:2021-02-15 06:50:08,617.617 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-15T06:50:08.679 INFO:teuthology.kill:2021-02-15 06:50:08,678.678 INFO:teuthology.orchestra.console:Power off smithi031 2021-02-15T06:50:08.697 INFO:teuthology.kill:2021-02-15 06:50:08,697.697 INFO:teuthology.orchestra.console:Power off smithi080 2021-02-15T06:50:08.852 INFO:teuthology.kill:2021-02-15 06:50:08,852.852 INFO:teuthology.orchestra.console:Power off for smithi031 completed 2021-02-15T06:50:08.856 INFO:teuthology.kill:2021-02-15 06:50:08,855.855 INFO:teuthology.orchestra.console:Power off for smithi080 completed 2021-02-15T06:50:08.999 INFO:teuthology.kill:2021-02-15 06:50:08,998.998 INFO:teuthology.lock.ops:unlocked smithi031.front.sepia.ceph.com 2021-02-15T06:50:09.002 INFO:teuthology.kill:2021-02-15 06:50:09,002.002 INFO:teuthology.lock.ops:unlocked smithi080.front.sepia.ceph.com 2021-02-15T06:52:09.235 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-02-15T06:52:09.464 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-02-15T06:52:09.464 INFO:teuthology.nuke:Checking targets against current locks 2021-02-15T06:52:09.573 INFO:teuthology.nuke:Not nuking smithi031.front.sepia.ceph.com because description doesn't match 2021-02-15T06:52:09.573 INFO:teuthology.nuke:Not nuking smithi080.front.sepia.ceph.com because description doesn't match