2021-02-14T18:24:20.227 INFO:root:teuthology version: 1.1.0-64d5f180 2021-02-14T18:24:20.353 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi081.front.sepia.ceph.com' reimaging 2021-02-14T18:24:20.354 INFO:teuthology.lock.ops:Updating [ubuntu@smithi081.front.sepia.ceph.com]: reset os type and version on server 2021-02-14T18:24:20.354 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2021-02-14T18:24:20.371 INFO:teuthology.lock.ops:Node 'ubuntu@smithi081.front.sepia.ceph.com' reimaging is complete 2021-02-14T18:24:20.371 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi200.front.sepia.ceph.com' reimaging 2021-02-14T18:24:20.371 INFO:teuthology.lock.ops:Updating [ubuntu@smithi200.front.sepia.ceph.com]: reset os type and version on server 2021-02-14T18:24:20.372 INFO:teuthology.lock.ops:Updating smithi200.front.sepia.ceph.com on lock server 2021-02-14T18:24:20.389 INFO:teuthology.lock.ops:Node 'ubuntu@smithi200.front.sepia.ceph.com' reimaging is complete 2021-02-14T18:24:20.595 INFO:teuthology.provision.fog.smithi200:Scheduling deploy of ubuntu 18.04 2021-02-14T18:24:20.605 INFO:teuthology.provision.fog.smithi081:Scheduling deploy of ubuntu 18.04 2021-02-14T18:24:21.005 INFO:teuthology.orchestra.console:Power off smithi200 2021-02-14T18:24:21.064 INFO:teuthology.orchestra.console:Power off smithi081 2021-02-14T18:24:29.376 INFO:teuthology.orchestra.console:Power off for smithi200 completed 2021-02-14T18:24:29.477 INFO:teuthology.orchestra.console:Power on smithi200 2021-02-14T18:24:33.541 INFO:teuthology.orchestra.console:Power off for smithi081 completed 2021-02-14T18:24:33.641 INFO:teuthology.orchestra.console:Power on smithi081 2021-02-14T18:24:37.840 INFO:teuthology.orchestra.console:Power on for smithi200 completed 2021-02-14T18:24:37.941 INFO:teuthology.provision.fog.smithi200:Waiting for deploy to finish 2021-02-14T18:24:42.006 INFO:teuthology.orchestra.console:Power on for smithi081 completed 2021-02-14T18:24:42.107 INFO:teuthology.provision.fog.smithi081:Waiting for deploy to finish 2021-02-14T18:27:28.821 ERROR:teuthology.orchestra.connection:Error authenticating with smithi081.front.sepia.ceph.com: Authentication failed. 2021-02-14T18:29:34.111 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-14T18:29:34.534 INFO:teuthology.provision.fog.smithi200:Node is ready 2021-02-14T18:29:34.581 INFO:teuthology.orchestra.run.smithi200.stdout:smithi200.front.sepia.ceph.com 2021-02-14T18:29:34.630 INFO:teuthology.orchestra.run.smithi200.stdout:172.21.15.200 smithi200.front.sepia.ceph.com smithi200 2021-02-14T18:29:34.973 INFO:teuthology.provision.fog.smithi200:Deploy complete! 2021-02-14T18:29:42.160 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-14T18:29:42.622 INFO:teuthology.provision.fog.smithi081:Node is ready 2021-02-14T18:29:42.671 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081.front.sepia.ceph.com 2021-02-14T18:29:42.720 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81 smithi081.front.sepia.ceph.com smithi081 2021-02-14T18:29:43.015 INFO:teuthology.provision.fog.smithi081:Deploy complete! 2021-02-14T18:29:43.162 INFO:teuthology.lock.ops:Checking smithi081.front.sepia.ceph.com 2021-02-14T18:29:43.162 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-14T18:29:43.178 INFO:teuthology.lock.ops:Checking smithi200.front.sepia.ceph.com 2021-02-14T18:29:43.178 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-14T18:29:43.191 INFO:teuthology.lock.ops:Updating [smithi081.front.sepia.ceph.com]: set os type and version on server 2021-02-14T18:29:43.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-14T18:29:43.491 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2021-02-14T18:29:43.541 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="Ubuntu" 2021-02-14T18:29:43.541 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-14T18:29:43.541 INFO:teuthology.orchestra.run.smithi081.stdout:ID=ubuntu 2021-02-14T18:29:43.542 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE=debian 2021-02-14T18:29:43.542 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-14T18:29:43.542 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="18.04" 2021-02-14T18:29:43.542 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-14T18:29:43.543 INFO:teuthology.orchestra.run.smithi081.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-14T18:29:43.543 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-14T18:29:43.543 INFO:teuthology.orchestra.run.smithi081.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-14T18:29:43.543 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_CODENAME=bionic 2021-02-14T18:29:43.543 INFO:teuthology.orchestra.run.smithi081.stdout:UBUNTU_CODENAME=bionic 2021-02-14T18:29:43.545 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2021-02-14T18:29:43.579 INFO:teuthology.lock.ops:Updating [smithi200.front.sepia.ceph.com]: set os type and version on server 2021-02-14T18:29:43.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-14T18:29:44.061 INFO:teuthology.orchestra.run.smithi200.stdout:x86_64 2021-02-14T18:29:44.116 INFO:teuthology.orchestra.run.smithi200.stdout:NAME="Ubuntu" 2021-02-14T18:29:44.116 INFO:teuthology.orchestra.run.smithi200.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-14T18:29:44.116 INFO:teuthology.orchestra.run.smithi200.stdout:ID=ubuntu 2021-02-14T18:29:44.116 INFO:teuthology.orchestra.run.smithi200.stdout:ID_LIKE=debian 2021-02-14T18:29:44.116 INFO:teuthology.orchestra.run.smithi200.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-14T18:29:44.116 INFO:teuthology.orchestra.run.smithi200.stdout:VERSION_ID="18.04" 2021-02-14T18:29:44.116 INFO:teuthology.orchestra.run.smithi200.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-14T18:29:44.116 INFO:teuthology.orchestra.run.smithi200.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-14T18:29:44.117 INFO:teuthology.orchestra.run.smithi200.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-14T18:29:44.117 INFO:teuthology.orchestra.run.smithi200.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-14T18:29:44.117 INFO:teuthology.orchestra.run.smithi200.stdout:VERSION_CODENAME=bionic 2021-02-14T18:29:44.117 INFO:teuthology.orchestra.run.smithi200.stdout:UBUNTU_CODENAME=bionic 2021-02-14T18:29:44.117 INFO:teuthology.lock.ops:Updating smithi200.front.sepia.ceph.com on lock server 2021-02-14T18:29:44.198 INFO:teuthology.dispatcher.supervisor:Running job 5881017 2021-02-14T18:29:44.198 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/virtualenv/bin/teuthology -v --owner scheduled_sseshasa@teuthology --archive /home/teuthworker/archive/sseshasa-2021-02-14_18:00:38-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5881017 --name sseshasa-2021-02-14_18:00:38-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi --description rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} -- /home/teuthworker/archive/sseshasa-2021-02-14_18:00:38-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5881017/orig.config.yaml 2021-02-14T18:29:44.202 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sseshasa-2021-02-14_18:00:38-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5881017 2021-02-14T18:29:44.202 INFO:teuthology.dispatcher.supervisor:Job PID: 15473 2021-02-14T18:29:44.202 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-02-15T06:30:00.732 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-02-15T06:30:01.040 INFO:teuthology.kill:Killing Pids: {15473} 2021-02-15T06:30:02.097 INFO:teuthology.kill:Nuking machines: ['smithi081', 'smithi200'] 2021-02-15T06:30:02.923 INFO:teuthology.kill:2021-02-15 06:30:02,922.922 INFO:teuthology.nuke:targets: 2021-02-15T06:30:02.923 INFO:teuthology.kill: smithi081.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDflUrlWD6qPsmqSlnd9OTBQrf54dEM4e0dbomK0fvCCokE5xFwUf4NzcKR/rbzmzY1NE2W8jr/xjTu5p9Dixeb5VEQ6galQFjgxM5R444Lpgxf795t4gRi5cP1MG3ZE7Ra9zQN9Lt9pKYdMUjjmopW7uLWYldzF0Xr7BzDgP54xGjPMC7ymd3tLUIuvoccmQKScDaFX4lyClN/ubDhQUd+4T3uQCFcp9A87k02Ak2jLh+FCTsQuWd+c9ImqCrKvzMVAKzpxqBSwhYH7DSfdVB9Mlb8F86kpeNjH+dHyFOShvBkq/kdM8nnFJPQZOKTxZjbxQPeliDqmPWMi1IVN3rv 2021-02-15T06:30:02.923 INFO:teuthology.kill: smithi200.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5mVlAE1LbGUoDICS388NJMgg/EqXbYX6ASB2ImUqFqDTIhNsaASajUPfOB2E3sDV5rJRnWJs0cxhJ6YiVaiguOiyv5DK4LXAvQr6y3llmTDskhPBMfK1ImGftoA1T1TnDViVMiXd1jNoR8P6vxl2+Mvlxv5XlU/sU0wS0wwtCjXFsOpJIDiBg4YhkL9kRRDqFm3TSj09pdBW9C3uLClTIZnBIPf2zADl1zMbRNxlc6/DoOss9f4Qernxipb4ht7A9C7qKODQbbdBHsniXHjmlENWoNwNWWZaedW85deoj41fmL2Hu2N3ULsJdjR2CnLuaXbIqn6L2/RdXQjaUiWEd 2021-02-15T06:30:02.923 INFO:teuthology.kill:2021-02-15 06:30:02,923.923 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-15T06:30:02.929 INFO:teuthology.kill:2021-02-15 06:30:02,929.929 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-15T06:30:02.988 INFO:teuthology.kill:2021-02-15 06:30:02,988.988 INFO:teuthology.orchestra.console:Power off smithi081 2021-02-15T06:30:03.002 INFO:teuthology.kill:2021-02-15 06:30:03,002.002 INFO:teuthology.orchestra.console:Power off smithi200 2021-02-15T06:30:11.356 INFO:teuthology.kill:2021-02-15 06:30:11,356.356 INFO:teuthology.orchestra.console:Power off for smithi081 completed 2021-02-15T06:30:15.476 INFO:teuthology.kill:2021-02-15 06:30:15,476.476 INFO:teuthology.orchestra.console:Power off for smithi200 completed 2021-02-15T06:30:15.704 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-02-15T06:30:15.704 INFO:teuthology.task.internal:roles: ubuntu@smithi200.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-02-15T06:30:15.704 INFO:teuthology.misc:Compressing logs... 2021-02-15T06:30:15.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-15T06:31:15.709 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/misc.py", line 1372, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-02-15T06:31:15.777 INFO:teuthology.kill:No teuthology processes running 2021-02-15T06:31:16.721 INFO:teuthology.kill:Nuking machines: ['smithi081', 'smithi200'] 2021-02-15T06:31:17.576 INFO:teuthology.kill:2021-02-15 06:31:17,575.575 INFO:teuthology.nuke:targets: 2021-02-15T06:31:17.576 INFO:teuthology.kill: smithi081.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDflUrlWD6qPsmqSlnd9OTBQrf54dEM4e0dbomK0fvCCokE5xFwUf4NzcKR/rbzmzY1NE2W8jr/xjTu5p9Dixeb5VEQ6galQFjgxM5R444Lpgxf795t4gRi5cP1MG3ZE7Ra9zQN9Lt9pKYdMUjjmopW7uLWYldzF0Xr7BzDgP54xGjPMC7ymd3tLUIuvoccmQKScDaFX4lyClN/ubDhQUd+4T3uQCFcp9A87k02Ak2jLh+FCTsQuWd+c9ImqCrKvzMVAKzpxqBSwhYH7DSfdVB9Mlb8F86kpeNjH+dHyFOShvBkq/kdM8nnFJPQZOKTxZjbxQPeliDqmPWMi1IVN3rv 2021-02-15T06:31:17.576 INFO:teuthology.kill: smithi200.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5mVlAE1LbGUoDICS388NJMgg/EqXbYX6ASB2ImUqFqDTIhNsaASajUPfOB2E3sDV5rJRnWJs0cxhJ6YiVaiguOiyv5DK4LXAvQr6y3llmTDskhPBMfK1ImGftoA1T1TnDViVMiXd1jNoR8P6vxl2+Mvlxv5XlU/sU0wS0wwtCjXFsOpJIDiBg4YhkL9kRRDqFm3TSj09pdBW9C3uLClTIZnBIPf2zADl1zMbRNxlc6/DoOss9f4Qernxipb4ht7A9C7qKODQbbdBHsniXHjmlENWoNwNWWZaedW85deoj41fmL2Hu2N3ULsJdjR2CnLuaXbIqn6L2/RdXQjaUiWEd 2021-02-15T06:31:17.599 INFO:teuthology.kill:2021-02-15 06:31:17,598.598 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-15T06:31:17.601 INFO:teuthology.kill:2021-02-15 06:31:17,601.601 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-15T06:31:17.659 INFO:teuthology.kill:2021-02-15 06:31:17,659.659 INFO:teuthology.orchestra.console:Power off smithi200 2021-02-15T06:31:17.672 INFO:teuthology.kill:2021-02-15 06:31:17,672.672 INFO:teuthology.orchestra.console:Power off smithi081 2021-02-15T06:31:17.801 INFO:teuthology.kill:2021-02-15 06:31:17,800.800 INFO:teuthology.orchestra.console:Power off for smithi200 completed 2021-02-15T06:31:17.812 INFO:teuthology.kill:2021-02-15 06:31:17,812.812 INFO:teuthology.orchestra.console:Power off for smithi081 completed 2021-02-15T06:31:17.944 INFO:teuthology.kill:2021-02-15 06:31:17,944.944 INFO:teuthology.lock.ops:unlocked smithi200.front.sepia.ceph.com 2021-02-15T06:31:17.949 INFO:teuthology.kill:2021-02-15 06:31:17,949.949 INFO:teuthology.lock.ops:unlocked smithi081.front.sepia.ceph.com 2021-02-15T06:33:18.132 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-02-15T06:33:18.236 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-02-15T06:33:18.237 INFO:teuthology.nuke:Checking targets against current locks 2021-02-15T06:33:18.328 INFO:teuthology.nuke:Not nuking smithi200.front.sepia.ceph.com because description doesn't match 2021-02-15T06:33:18.328 INFO:teuthology.nuke:Not nuking smithi081.front.sepia.ceph.com because description doesn't match