2020-11-23T13:49:40.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:49:49.289 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:49:51.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:49:58.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:00.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:07.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:10.056 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:16.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:19.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:25.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:28.172 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:34.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:37.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:46.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:55.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:11.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:18.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:18.075 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-23T13:51:24.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:24.092 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-23T13:51:30.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:30.106 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-23T13:51:32.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:32.982 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-23T13:51:36.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:37.150 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-23T13:51:38.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:38.996 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-23T13:51:43.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:43.314 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-23T13:51:43.315 INFO:teuthology.orchestra.run.smithi123:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-23T13:51:43.704 INFO:teuthology.provision.fog.smithi123:Node is ready 2020-11-23T13:51:43.705 INFO:teuthology.orchestra.run.smithi123:> hostname 2020-11-23T13:51:43.753 INFO:teuthology.orchestra.run.smithi123.stdout:smithi123.front.sepia.ceph.com 2020-11-23T13:51:43.754 INFO:teuthology.orchestra.run.smithi123:> grep smithi123.front.sepia.ceph.com /etc/hosts 2020-11-23T13:51:43.802 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123 smithi123.front.sepia.ceph.com smithi123 2020-11-23T13:51:43.802 INFO:teuthology.orchestra.run.smithi123:> sudo hostname smithi123 2020-11-23T13:51:43.987 INFO:teuthology.orchestra.run.smithi123:> sudo sed -i -e 's/smithi123.front.sepia.ceph.com/smithi123/g' /etc/hosts 2020-11-23T13:51:44.002 INFO:teuthology.orchestra.run.smithi123:> sudo sed -i -e 's/172.21.15.123/172.21.15.123/g' /etc/hosts 2020-11-23T13:51:44.059 INFO:teuthology.orchestra.run.smithi123:> sudo sed -i -e 's/smithi123.front.sepia.ceph.com/smithi123/g' /etc/hostname 2020-11-23T13:51:44.114 INFO:teuthology.orchestra.run.smithi123:> sudo hostname smithi123 2020-11-23T13:51:44.170 INFO:teuthology.provision.fog.smithi123:Deploy complete! 2020-11-23T13:51:44.170 DEBUG:teuthology.parallel:result is None 2020-11-23T13:51:44.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:45.007 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-23T13:51:51.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:51.020 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-23T13:51:57.021 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:57.032 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-23T13:52:03.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:52:03.111 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-23T13:52:03.111 INFO:teuthology.orchestra.run.smithi146:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-23T13:52:03.487 INFO:teuthology.provision.fog.smithi146:Node is ready 2020-11-23T13:52:03.487 INFO:teuthology.orchestra.run.smithi146:> hostname 2020-11-23T13:52:03.533 INFO:teuthology.orchestra.run.smithi146.stdout:smithi146.front.sepia.ceph.com 2020-11-23T13:52:03.533 INFO:teuthology.orchestra.run.smithi146:> grep smithi146.front.sepia.ceph.com /etc/hosts 2020-11-23T13:52:03.581 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146 smithi146.front.sepia.ceph.com smithi146 2020-11-23T13:52:03.582 INFO:teuthology.orchestra.run.smithi146:> sudo hostname smithi146 2020-11-23T13:52:03.768 INFO:teuthology.orchestra.run.smithi146:> sudo sed -i -e 's/smithi146.front.sepia.ceph.com/smithi146/g' /etc/hosts 2020-11-23T13:52:03.783 INFO:teuthology.orchestra.run.smithi146:> sudo sed -i -e 's/172.21.15.146/172.21.15.146/g' /etc/hosts 2020-11-23T13:52:03.841 INFO:teuthology.orchestra.run.smithi146:> sudo sed -i -e 's/smithi146.front.sepia.ceph.com/smithi146/g' /etc/hostname 2020-11-23T13:52:03.898 INFO:teuthology.orchestra.run.smithi146:> sudo hostname smithi146 2020-11-23T13:52:03.952 INFO:teuthology.provision.fog.smithi146:Deploy complete! 2020-11-23T13:52:03.953 DEBUG:teuthology.parallel:result is None 2020-11-23T13:52:04.075 INFO:teuthology.lock.ops:Checking smithi146.front.sepia.ceph.com 2020-11-23T13:52:04.076 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-23T13:52:04.091 INFO:teuthology.lock.ops:Checking smithi123.front.sepia.ceph.com 2020-11-23T13:52:04.091 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-23T13:52:04.105 INFO:teuthology.lock.ops:Updating [smithi146.front.sepia.ceph.com]: set os type and version on server 2020-11-23T13:52:04.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-23T13:52:04.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:52:04.179 INFO:teuthology.orchestra.run.smithi146:> true 2020-11-23T13:52:04.353 INFO:teuthology.orchestra.run.smithi146:> uname -m 2020-11-23T13:52:04.401 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2020-11-23T13:52:04.401 INFO:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2020-11-23T13:52:04.449 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="Ubuntu" 2020-11-23T13:52:04.449 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-23T13:52:04.449 INFO:teuthology.orchestra.run.smithi146.stdout:ID=ubuntu 2020-11-23T13:52:04.450 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE=debian 2020-11-23T13:52:04.450 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-23T13:52:04.450 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="18.04" 2020-11-23T13:52:04.450 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-23T13:52:04.450 INFO:teuthology.orchestra.run.smithi146.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-23T13:52:04.451 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-23T13:52:04.451 INFO:teuthology.orchestra.run.smithi146.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-23T13:52:04.451 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_CODENAME=bionic 2020-11-23T13:52:04.451 INFO:teuthology.orchestra.run.smithi146.stdout:UBUNTU_CODENAME=bionic 2020-11-23T13:52:04.452 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2020-11-23T13:52:04.466 INFO:teuthology.lock.ops:Updating [smithi123.front.sepia.ceph.com]: set os type and version on server 2020-11-23T13:52:04.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-23T13:52:04.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:52:04.539 INFO:teuthology.orchestra.run.smithi123:> true 2020-11-23T13:52:04.732 INFO:teuthology.orchestra.run.smithi123:> uname -m 2020-11-23T13:52:04.778 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2020-11-23T13:52:04.778 INFO:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2020-11-23T13:52:04.823 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="Ubuntu" 2020-11-23T13:52:04.824 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-23T13:52:04.824 INFO:teuthology.orchestra.run.smithi123.stdout:ID=ubuntu 2020-11-23T13:52:04.824 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE=debian 2020-11-23T13:52:04.824 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-23T13:52:04.824 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="18.04" 2020-11-23T13:52:04.825 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-23T13:52:04.825 INFO:teuthology.orchestra.run.smithi123.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-23T13:52:04.825 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-23T13:52:04.825 INFO:teuthology.orchestra.run.smithi123.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-23T13:52:04.825 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_CODENAME=bionic 2020-11-23T13:52:04.826 INFO:teuthology.orchestra.run.smithi123.stdout:UBUNTU_CODENAME=bionic 2020-11-23T13:52:04.826 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2020-11-23T13:52:04.850 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-11-23T13:52:04.885 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi123.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDxh4x6WmmCCJiTdJ0MXBZwI+WXIXphrMnpeGWm4H5o1u4wL/qh5+q6BF11kcH14xjBYTYX0E1tg+qYzufrdER9QemtPGjNTtGKzhFUtmnRZYQMqiw2sDODO+ZK50ZzFhLEjIPG+XaEOkaMjubJztgJ33vitluYydUzcjkPz/P/uAox48Lpi8Z5sOA04l08zrfXn81w4OU0Jf4kqTkqQm+aC4Vac2guHQgBwOLFhibpBcm2yJ/1YKNKtGoFTdO08oZNV0+JgGtqhfAdocK3cMm0dCHIvvwzGnEUODWg2SlMrPW8804DCWLllOqW3SeoUlako/RMqVkFXu7uk9HU4P8J smithi146.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDD/GJOJHsmQsJ3OETCX/e7ZiXOrhn1t7CnU9W242XFD+tE8zo0OLw6vj4rSmV/m4d/srLOg7ki1yZ5ELQlADpdZPatia7fDPpo0JiVlwAHm7X0/ktQW/CHCPQ5Q6qwgMAoh72w8EAT7jJtMhwaTS1Eb6yL4/mbRtCQ2C6RUVV1qQTojcH+YzHRjEtqMZ9cJRliyKFYLIf0YNdhFIbwfGsXuEyLVbtq7kGGMMN3mqGfAOArVdgHxjvinZ7h2fAIadxUdhs/8V7minQXazKvtTq+Suz/16nNpLf1BlXI2dveu4TYMM4DCNs4zuaA0WHW4doTxvs09k1J7kS1LT/o+xJX 2020-11-23T13:52:04.885 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-23T13:52:04.933 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-23T13:52:05.004 ERROR:teuthology.timer:Failed to write timing.yaml ! Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/timer.py", line 112, in write yaml.safe_dump(self.data, f, default_flow_style=False) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 28, in represent self.serialize(node) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/serializer.py", line 55, in serialize self.emit(DocumentEndEvent(explicit=self.use_explicit_end)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 115, in emit self.state() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 220, in expect_document_end self.flush_stream() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 790, in flush_stream self.stream.flush() OSError: [Errno 28] No space left on device 2020-11-23T13:52:05.005 INFO:teuthology.task.internal:Saving configuration 2020-11-23T13:52:05.017 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-23T13:52:05.030 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-23T13:52:05.046 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-23 13:46:42.753016', 'description': '/home/teuthworker/archive/teuthology-2020-11-23_03:05:03-rgw-master-distro-basic-smithi/5647469', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEbky9tFXlOl4nsIPzjmq4u+tmzXmomKgCFgq7zF5eoX', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi146.front.sepia.ceph.com'} 2020-11-23T13:52:05.060 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-23 13:46:42.753037', 'description': '/home/teuthworker/archive/teuthology-2020-11-23_03:05:03-rgw-master-distro-basic-smithi/5647469', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMx0vnQ35u5enF4ABsU1rZoyqS8p93DUb4Fo246qtwfn', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi123.front.sepia.ceph.com'} 2020-11-23T13:52:05.061 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-23T13:52:05.073 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2020-11-23T13:52:05.073 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2020-11-23T13:52:05.073 INFO:teuthology.run_tasks:Running task console_log... 2020-11-23T13:52:05.137 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5db3b4f0d0>, signals=[15]) 2020-11-23T13:52:05.138 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-23T13:52:05.148 INFO:teuthology.task.internal:Opening connections... 2020-11-23T13:52:05.148 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2020-11-23T13:52:05.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:52:05.223 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2020-11-23T13:52:05.225 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:52:05.298 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-23T13:52:05.306 INFO:teuthology.orchestra.run.smithi146:> uname -m 2020-11-23T13:52:05.413 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2020-11-23T13:52:05.413 INFO:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2020-11-23T13:52:05.460 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="Ubuntu" 2020-11-23T13:52:05.460 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-23T13:52:05.461 INFO:teuthology.orchestra.run.smithi146.stdout:ID=ubuntu 2020-11-23T13:52:05.461 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE=debian 2020-11-23T13:52:05.461 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-23T13:52:05.461 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="18.04" 2020-11-23T13:52:05.461 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-23T13:52:05.462 INFO:teuthology.orchestra.run.smithi146.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-23T13:52:05.462 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-23T13:52:05.462 INFO:teuthology.orchestra.run.smithi146.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-23T13:52:05.462 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_CODENAME=bionic 2020-11-23T13:52:05.462 INFO:teuthology.orchestra.run.smithi146.stdout:UBUNTU_CODENAME=bionic 2020-11-23T13:52:05.463 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2020-11-23T13:52:05.481 INFO:teuthology.orchestra.run.smithi123:> uname -m 2020-11-23T13:52:05.489 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2020-11-23T13:52:05.490 INFO:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2020-11-23T13:52:05.535 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="Ubuntu" 2020-11-23T13:52:05.536 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-23T13:52:05.536 INFO:teuthology.orchestra.run.smithi123.stdout:ID=ubuntu 2020-11-23T13:52:05.536 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE=debian 2020-11-23T13:52:05.536 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-23T13:52:05.537 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="18.04" 2020-11-23T13:52:05.537 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-23T13:52:05.537 INFO:teuthology.orchestra.run.smithi123.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-23T13:52:05.537 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-23T13:52:05.538 INFO:teuthology.orchestra.run.smithi123.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-23T13:52:05.538 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_CODENAME=bionic 2020-11-23T13:52:05.538 INFO:teuthology.orchestra.run.smithi123.stdout:UBUNTU_CODENAME=bionic 2020-11-23T13:52:05.538 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2020-11-23T13:52:05.551 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-23T13:52:05.576 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 90, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 69, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 251, in serialize_remote_roles yaml.safe_dump(info_yaml, info_file, default_flow_style=False) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 28, in represent self.serialize(node) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/serializer.py", line 55, in serialize self.emit(DocumentEndEvent(explicit=self.use_explicit_end)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 115, in emit self.state() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 220, in expect_document_end self.flush_stream() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 790, in flush_stream self.stream.flush() OSError: [Errno 28] No space left on device 2020-11-23T13:52:05.620 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=58dad8b086d546e48e430f5964c54e89 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 90, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 69, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 251, in serialize_remote_roles yaml.safe_dump(info_yaml, info_file, default_flow_style=False) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 28, in represent self.serialize(node) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/serializer.py", line 55, in serialize self.emit(DocumentEndEvent(explicit=self.use_explicit_end)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 115, in emit self.state() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 220, in expect_document_end self.flush_stream() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 790, in flush_stream self.stream.flush() OSError: [Errno 28] No space left on device 2020-11-23T13:52:05.620 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2020-11-23T13:52:05.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.lock_machines 2020-11-23T13:52:05.655 INFO:teuthology.nuke:Checking targets against current locks 2020-11-23T13:52:05.747 DEBUG:teuthology.nuke:shortname: smithi123 2020-11-23T13:52:05.747 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-23T13:52:05.750 DEBUG:teuthology.nuke:shortname: smithi146 2020-11-23T13:52:05.750 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-23T13:52:05.766 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-23 13:46:42.753037', 'description': '/home/teuthworker/archive/teuthology-2020-11-23_03:05:03-rgw-master-distro-basic-smithi/5647469', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDxh4x6WmmCCJiTdJ0MXBZwI+WXIXphrMnpeGWm4H5o1u4wL/qh5+q6BF11kcH14xjBYTYX0E1tg+qYzufrdER9QemtPGjNTtGKzhFUtmnRZYQMqiw2sDODO+ZK50ZzFhLEjIPG+XaEOkaMjubJztgJ33vitluYydUzcjkPz/P/uAox48Lpi8Z5sOA04l08zrfXn81w4OU0Jf4kqTkqQm+aC4Vac2guHQgBwOLFhibpBcm2yJ/1YKNKtGoFTdO08oZNV0+JgGtqhfAdocK3cMm0dCHIvvwzGnEUODWg2SlMrPW8804DCWLllOqW3SeoUlako/RMqVkFXu7uk9HU4P8J', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi123.front.sepia.ceph.com'} 2020-11-23T13:52:05.768 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-23 13:46:42.753016', 'description': '/home/teuthworker/archive/teuthology-2020-11-23_03:05:03-rgw-master-distro-basic-smithi/5647469', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDD/GJOJHsmQsJ3OETCX/e7ZiXOrhn1t7CnU9W242XFD+tE8zo0OLw6vj4rSmV/m4d/srLOg7ki1yZ5ELQlADpdZPatia7fDPpo0JiVlwAHm7X0/ktQW/CHCPQ5Q6qwgMAoh72w8EAT7jJtMhwaTS1Eb6yL4/mbRtCQ2C6RUVV1qQTojcH+YzHRjEtqMZ9cJRliyKFYLIf0YNdhFIbwfGsXuEyLVbtq7kGGMMN3mqGfAOArVdgHxjvinZ7h2fAIadxUdhs/8V7minQXazKvtTq+Suz/16nNpLf1BlXI2dveu4TYMM4DCNs4zuaA0WHW4doTxvs09k1J7kS1LT/o+xJX', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi146.front.sepia.ceph.com'} 2020-11-23T13:52:05.806 INFO:teuthology.orchestra.console:Power off smithi123 2020-11-23T13:52:05.807 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-23T13:52:05.819 INFO:teuthology.orchestra.console:Power off smithi146 2020-11-23T13:52:05.820 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-23T13:52:05.827 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-23T13:52:05.839 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-23T13:52:09.850 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-23T13:52:09.859 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status