2020-11-23T13:49:46.534 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:49:46.635 ERROR:teuthology.orchestra.connection:Error connecting to smithi019.front.sepia.ceph.com Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 105, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 446, in connect passphrase, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 764, in _auth raise saved_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 740, in _auth self._transport.auth_publickey(username, key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 1580, in auth_publickey return self.auth_handler.wait_for_response(my_event) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/auth_handler.py", line 250, in wait_for_response raise e paramiko.ssh_exception.AuthenticationException: Authentication failed. 2020-11-23T13:49:57.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:06.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:15.622 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:24.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:33.734 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:42.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:51.878 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:50:53.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:00.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:11.686 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:18.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:18.426 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-23T13:51:18.428 INFO:teuthology.orchestra.run.smithi019:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-23T13:51:20.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:28.782 INFO:teuthology.provision.fog.smithi019:Node is ready 2020-11-23T13:51:28.782 INFO:teuthology.orchestra.run.smithi019:> hostname 2020-11-23T13:51:28.799 INFO:teuthology.orchestra.run.smithi019.stdout:smithi179 2020-11-23T13:51:28.799 INFO:teuthology.orchestra.run.smithi019:> grep smithi179 /etc/hosts 2020-11-23T13:51:28.853 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.179 smithi179.front.sepia.ceph.com smithi179 2020-11-23T13:51:28.853 INFO:teuthology.orchestra.run.smithi019:> sudo hostname smithi019 2020-11-23T13:51:28.984 INFO:teuthology.orchestra.run.smithi019:> sudo sed -i -e 's/smithi179/smithi019/g' /etc/hosts 2020-11-23T13:51:29.016 INFO:teuthology.orchestra.run.smithi019:> sudo sed -i -e 's/172.21.15.179/172.21.15.19/g' /etc/hosts 2020-11-23T13:51:29.086 INFO:teuthology.orchestra.run.smithi019:> sudo sed -i -e 's/smithi179/smithi019/g' /etc/hostname 2020-11-23T13:51:29.157 INFO:teuthology.orchestra.run.smithi019:> sudo hostname smithi019 2020-11-23T13:51:29.227 INFO:teuthology.provision.fog.smithi019:Deploy complete! 2020-11-23T13:51:29.228 DEBUG:teuthology.parallel:result is None 2020-11-23T13:51:45.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:54.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:51:56.569 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-23T13:51:56.569 INFO:teuthology.orchestra.run.smithi025:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-23T13:52:06.982 INFO:teuthology.provision.fog.smithi025:Node is ready 2020-11-23T13:52:06.983 INFO:teuthology.orchestra.run.smithi025:> hostname 2020-11-23T13:52:07.041 INFO:teuthology.orchestra.run.smithi025.stdout:smithi179 2020-11-23T13:52:07.041 INFO:teuthology.orchestra.run.smithi025:> grep smithi179 /etc/hosts 2020-11-23T13:52:07.097 INFO:teuthology.orchestra.run.smithi025.stdout:172.21.15.179 smithi179.front.sepia.ceph.com smithi179 2020-11-23T13:52:07.097 INFO:teuthology.orchestra.run.smithi025:> sudo hostname smithi025 2020-11-23T13:52:07.239 INFO:teuthology.orchestra.run.smithi025:> sudo sed -i -e 's/smithi179/smithi025/g' /etc/hosts 2020-11-23T13:52:07.273 INFO:teuthology.orchestra.run.smithi025:> sudo sed -i -e 's/172.21.15.179/172.21.15.25/g' /etc/hosts 2020-11-23T13:52:07.356 INFO:teuthology.orchestra.run.smithi025:> sudo sed -i -e 's/smithi179/smithi025/g' /etc/hostname 2020-11-23T13:52:07.433 INFO:teuthology.orchestra.run.smithi025:> sudo hostname smithi025 2020-11-23T13:52:07.503 INFO:teuthology.provision.fog.smithi025:Deploy complete! 2020-11-23T13:52:07.504 DEBUG:teuthology.parallel:result is None 2020-11-23T13:52:07.618 INFO:teuthology.lock.ops:Checking smithi025.front.sepia.ceph.com 2020-11-23T13:52:07.618 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-23T13:52:07.634 INFO:teuthology.lock.ops:Checking smithi019.front.sepia.ceph.com 2020-11-23T13:52:07.634 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-23T13:52:07.648 INFO:teuthology.lock.ops:Updating [smithi025.front.sepia.ceph.com]: set os type and version on server 2020-11-23T13:52:07.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-23T13:52:07.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:52:07.715 INFO:teuthology.orchestra.run.smithi025:> true 2020-11-23T13:52:07.891 INFO:teuthology.orchestra.run.smithi025:> uname -m 2020-11-23T13:52:07.945 INFO:teuthology.orchestra.run.smithi025.stdout:x86_64 2020-11-23T13:52:07.945 INFO:teuthology.orchestra.run.smithi025:> cat /etc/os-release 2020-11-23T13:52:07.998 INFO:teuthology.orchestra.run.smithi025.stdout:NAME="CentOS Linux" 2020-11-23T13:52:07.999 INFO:teuthology.orchestra.run.smithi025.stdout:VERSION="8 (Core)" 2020-11-23T13:52:07.999 INFO:teuthology.orchestra.run.smithi025.stdout:ID="centos" 2020-11-23T13:52:07.999 INFO:teuthology.orchestra.run.smithi025.stdout:ID_LIKE="rhel fedora" 2020-11-23T13:52:07.999 INFO:teuthology.orchestra.run.smithi025.stdout:VERSION_ID="8" 2020-11-23T13:52:08.000 INFO:teuthology.orchestra.run.smithi025.stdout:PLATFORM_ID="platform:el8" 2020-11-23T13:52:08.000 INFO:teuthology.orchestra.run.smithi025.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-23T13:52:08.000 INFO:teuthology.orchestra.run.smithi025.stdout:ANSI_COLOR="0;31" 2020-11-23T13:52:08.000 INFO:teuthology.orchestra.run.smithi025.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-23T13:52:08.000 INFO:teuthology.orchestra.run.smithi025.stdout:HOME_URL="https://www.centos.org/" 2020-11-23T13:52:08.000 INFO:teuthology.orchestra.run.smithi025.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-23T13:52:08.001 INFO:teuthology.orchestra.run.smithi025.stdout: 2020-11-23T13:52:08.001 INFO:teuthology.orchestra.run.smithi025.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-23T13:52:08.001 INFO:teuthology.orchestra.run.smithi025.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-23T13:52:08.001 INFO:teuthology.orchestra.run.smithi025.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-23T13:52:08.001 INFO:teuthology.orchestra.run.smithi025.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-23T13:52:08.002 INFO:teuthology.orchestra.run.smithi025.stdout: 2020-11-23T13:52:08.003 INFO:teuthology.lock.ops:Updating smithi025.front.sepia.ceph.com on lock server 2020-11-23T13:52:08.018 INFO:teuthology.lock.ops:Updating [smithi019.front.sepia.ceph.com]: set os type and version on server 2020-11-23T13:52:08.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-23T13:52:08.019 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:52:08.086 INFO:teuthology.orchestra.run.smithi019:> true 2020-11-23T13:52:08.271 INFO:teuthology.orchestra.run.smithi019:> uname -m 2020-11-23T13:52:08.325 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2020-11-23T13:52:08.326 INFO:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2020-11-23T13:52:08.379 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Linux" 2020-11-23T13:52:08.380 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="8 (Core)" 2020-11-23T13:52:08.380 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2020-11-23T13:52:08.380 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2020-11-23T13:52:08.380 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="8" 2020-11-23T13:52:08.381 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el8" 2020-11-23T13:52:08.381 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-23T13:52:08.381 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2020-11-23T13:52:08.381 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-23T13:52:08.381 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://www.centos.org/" 2020-11-23T13:52:08.382 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-23T13:52:08.382 INFO:teuthology.orchestra.run.smithi019.stdout: 2020-11-23T13:52:08.382 INFO:teuthology.orchestra.run.smithi019.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-23T13:52:08.382 INFO:teuthology.orchestra.run.smithi019.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-23T13:52:08.382 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-23T13:52:08.383 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-23T13:52:08.383 INFO:teuthology.orchestra.run.smithi019.stdout: 2020-11-23T13:52:08.383 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2020-11-23T13:52:08.399 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-11-23T13:52:08.433 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi019.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzS2w1uPBgzzeaXu3fzhtaXK3DMK7Wk+CE/+SbQ+Zg+Z4IeFqAF0GcZbtL0Z1laJqOZyOWyFJ5MCOCo2gSFzhfLqjSu3VtcJNZXQXB+8Kyd3elbgJPTolZ52s4OBP6FR4krc5P0GQhEWiSG7uAcwgl1QIFI9UiY8zvieNdo/zRIpIlpQdTEo1eNmzRmV/GbDj1l55dtLLT4yjxPX8a8yuvY+BfLEVUJAK52kkX5+tqGiLlQWKWf+DF35p8GopTbdLk/HwX+mpn3EIl22R+XCwjHFyUwXyCT9mwW3Et5A3QCDH/75pCdyx01PwRguj0lDNYlPxTxUVGij+snjsHMdEOetpD+VMKGEmKHvxkPeShxEEdX0FrX5RuEaUfZw/Om7AIImOsODO8yiP83GC4JxCaYuFZjxtqwhW7sEgJYANGWJqVPWVCpSy3kF4XDKqpeYzLkCQEoU/Ccv+98qYS3VqRSHghAbPgAb+++eVfvatsOo36V0DM52KG0xZGAua0Nb0= smithi025.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmGSrisN45ZUKwTywL0kQftBcgMS8eGWzfEbwe9YfS6/JjXjLVwcuS5lQxtcvs1k90ZaVdI+xjuga/Tyw0ngXYs7SVqdZyaHdfRJtaKrWwVcJ7KL6XcAGi48XFPZMxiYqD19XNuoALZT1cQv7RfdIlJ+8ytzIpJcrL12yvJi2lm9PUriaSr1bUNafGOaGP5/hlr1xDjwu71m/ZYJ5wV+Q/TR0JR7krTi55eQwwWAWO1dmK+JgRkHVijLVD9HXkE7zO/M6jT0ASgKBUVmQk4wJs6FoKUuWB0MLoDD+DQC8zFLT7gRNimKgc3ZDcYHchTA7k+EKoatflaGcaLNU2MQNr0D62OjER++JXj3bKG7rN0j0/My1FQ0PPdvKSsWeIpKxc08rsKHnjvx9GtsX7wihaP7M6wQlLswtrEdS4Bi9aU8pAD1GkYgMBFIZxCJijuXOMYPQbqTbVqa1AVVBqPboRgvac0B4oDeSj5fU/7iv/sT0gCHGD+Oy5jQcKGCnMBtk= 2020-11-23T13:52:08.433 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-23T13:52:08.490 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-23T13:52:08.546 ERROR:teuthology.timer:Failed to write timing.yaml ! Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/timer.py", line 112, in write yaml.safe_dump(self.data, f, default_flow_style=False) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 28, in represent self.serialize(node) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/serializer.py", line 55, in serialize self.emit(DocumentEndEvent(explicit=self.use_explicit_end)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 115, in emit self.state() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 220, in expect_document_end self.flush_stream() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 790, in flush_stream self.stream.flush() OSError: [Errno 28] No space left on device 2020-11-23T13:52:08.548 INFO:teuthology.task.internal:Saving configuration 2020-11-23T13:52:08.563 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-23T13:52:08.576 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-23T13:52:08.593 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-23 13:47:03.695797', 'description': '/home/teuthworker/archive/teuthology-2020-11-23_03:05:03-rgw-master-distro-basic-smithi/5647472', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMzsCdJqC/R2rl2rPErJL6MhcjlSqaQtUdu734oPOPtiEiK5SF0xYWRXB4F7lP1Y2cE6tAJlNGr3XVqg76r1ZXc=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi025.front.sepia.ceph.com'} 2020-11-23T13:52:08.609 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-23 13:47:03.695819', 'description': '/home/teuthworker/archive/teuthology-2020-11-23_03:05:03-rgw-master-distro-basic-smithi/5647472', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMsMhWt+GNKdNEBL4SQk5pmVsvO0x6+45OA3LeByxIVWALzFtxacaO0B7FKzpfXPOiKiXHUkhcOerJvkTmVM18Y=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi019.front.sepia.ceph.com'} 2020-11-23T13:52:08.610 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-23T13:52:08.620 INFO:teuthology.task.internal:roles: ubuntu@smithi025.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2020-11-23T13:52:08.620 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2020-11-23T13:52:08.621 INFO:teuthology.run_tasks:Running task console_log... 2020-11-23T13:52:08.728 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbce96e3730>, signals=[15]) 2020-11-23T13:52:08.729 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-23T13:52:08.817 INFO:teuthology.task.internal:Opening connections... 2020-11-23T13:52:08.818 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi025.front.sepia.ceph.com 2020-11-23T13:52:08.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:52:08.898 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi019.front.sepia.ceph.com 2020-11-23T13:52:08.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-23T13:52:08.971 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-23T13:52:08.980 INFO:teuthology.orchestra.run.smithi025:> uname -m 2020-11-23T13:52:09.041 INFO:teuthology.orchestra.run.smithi025.stdout:x86_64 2020-11-23T13:52:09.041 INFO:teuthology.orchestra.run.smithi025:> cat /etc/os-release 2020-11-23T13:52:09.095 INFO:teuthology.orchestra.run.smithi025.stdout:NAME="CentOS Linux" 2020-11-23T13:52:09.096 INFO:teuthology.orchestra.run.smithi025.stdout:VERSION="8 (Core)" 2020-11-23T13:52:09.096 INFO:teuthology.orchestra.run.smithi025.stdout:ID="centos" 2020-11-23T13:52:09.096 INFO:teuthology.orchestra.run.smithi025.stdout:ID_LIKE="rhel fedora" 2020-11-23T13:52:09.096 INFO:teuthology.orchestra.run.smithi025.stdout:VERSION_ID="8" 2020-11-23T13:52:09.097 INFO:teuthology.orchestra.run.smithi025.stdout:PLATFORM_ID="platform:el8" 2020-11-23T13:52:09.097 INFO:teuthology.orchestra.run.smithi025.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-23T13:52:09.097 INFO:teuthology.orchestra.run.smithi025.stdout:ANSI_COLOR="0;31" 2020-11-23T13:52:09.097 INFO:teuthology.orchestra.run.smithi025.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-23T13:52:09.098 INFO:teuthology.orchestra.run.smithi025.stdout:HOME_URL="https://www.centos.org/" 2020-11-23T13:52:09.098 INFO:teuthology.orchestra.run.smithi025.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-23T13:52:09.098 INFO:teuthology.orchestra.run.smithi025.stdout: 2020-11-23T13:52:09.098 INFO:teuthology.orchestra.run.smithi025.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-23T13:52:09.098 INFO:teuthology.orchestra.run.smithi025.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-23T13:52:09.099 INFO:teuthology.orchestra.run.smithi025.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-23T13:52:09.099 INFO:teuthology.orchestra.run.smithi025.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-23T13:52:09.099 INFO:teuthology.orchestra.run.smithi025.stdout: 2020-11-23T13:52:09.100 INFO:teuthology.lock.ops:Updating smithi025.front.sepia.ceph.com on lock server 2020-11-23T13:52:09.119 INFO:teuthology.orchestra.run.smithi019:> uname -m 2020-11-23T13:52:09.134 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2020-11-23T13:52:09.135 INFO:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2020-11-23T13:52:09.189 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Linux" 2020-11-23T13:52:09.189 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="8 (Core)" 2020-11-23T13:52:09.189 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2020-11-23T13:52:09.189 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2020-11-23T13:52:09.189 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="8" 2020-11-23T13:52:09.190 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el8" 2020-11-23T13:52:09.190 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-23T13:52:09.190 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2020-11-23T13:52:09.190 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-23T13:52:09.190 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://www.centos.org/" 2020-11-23T13:52:09.191 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-23T13:52:09.191 INFO:teuthology.orchestra.run.smithi019.stdout: 2020-11-23T13:52:09.191 INFO:teuthology.orchestra.run.smithi019.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-23T13:52:09.191 INFO:teuthology.orchestra.run.smithi019.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-23T13:52:09.191 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-23T13:52:09.192 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-23T13:52:09.192 INFO:teuthology.orchestra.run.smithi019.stdout: 2020-11-23T13:52:09.192 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2020-11-23T13:52:09.207 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-23T13:52:09.231 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 90, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 69, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 251, in serialize_remote_roles yaml.safe_dump(info_yaml, info_file, default_flow_style=False) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 28, in represent self.serialize(node) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/serializer.py", line 55, in serialize self.emit(DocumentEndEvent(explicit=self.use_explicit_end)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 115, in emit self.state() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 220, in expect_document_end self.flush_stream() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 790, in flush_stream self.stream.flush() OSError: [Errno 28] No space left on device 2020-11-23T13:52:09.250 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e7541260e1ce48639833d9a1816387ba Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 90, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 69, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 251, in serialize_remote_roles yaml.safe_dump(info_yaml, info_file, default_flow_style=False) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 28, in represent self.serialize(node) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/serializer.py", line 55, in serialize self.emit(DocumentEndEvent(explicit=self.use_explicit_end)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 115, in emit self.state() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 220, in expect_document_end self.flush_stream() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/emitter.py", line 790, in flush_stream self.stream.flush() OSError: [Errno 28] No space left on device 2020-11-23T13:52:09.251 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2020-11-23T13:52:09.259 DEBUG:teuthology.run_tasks:Unwinding manager internal.lock_machines 2020-11-23T13:52:09.315 INFO:teuthology.nuke:Checking targets against current locks 2020-11-23T13:52:09.430 DEBUG:teuthology.nuke:shortname: smithi025 2020-11-23T13:52:09.430 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-23T13:52:09.433 DEBUG:teuthology.nuke:shortname: smithi019 2020-11-23T13:52:09.433 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-23T13:52:09.449 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-23 13:47:03.695797', 'description': '/home/teuthworker/archive/teuthology-2020-11-23_03:05:03-rgw-master-distro-basic-smithi/5647472', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmGSrisN45ZUKwTywL0kQftBcgMS8eGWzfEbwe9YfS6/JjXjLVwcuS5lQxtcvs1k90ZaVdI+xjuga/Tyw0ngXYs7SVqdZyaHdfRJtaKrWwVcJ7KL6XcAGi48XFPZMxiYqD19XNuoALZT1cQv7RfdIlJ+8ytzIpJcrL12yvJi2lm9PUriaSr1bUNafGOaGP5/hlr1xDjwu71m/ZYJ5wV+Q/TR0JR7krTi55eQwwWAWO1dmK+JgRkHVijLVD9HXkE7zO/M6jT0ASgKBUVmQk4wJs6FoKUuWB0MLoDD+DQC8zFLT7gRNimKgc3ZDcYHchTA7k+EKoatflaGcaLNU2MQNr0D62OjER++JXj3bKG7rN0j0/My1FQ0PPdvKSsWeIpKxc08rsKHnjvx9GtsX7wihaP7M6wQlLswtrEdS4Bi9aU8pAD1GkYgMBFIZxCJijuXOMYPQbqTbVqa1AVVBqPboRgvac0B4oDeSj5fU/7iv/sT0gCHGD+Oy5jQcKGCnMBtk=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi025.front.sepia.ceph.com'} 2020-11-23T13:52:09.452 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-23 13:47:03.695819', 'description': '/home/teuthworker/archive/teuthology-2020-11-23_03:05:03-rgw-master-distro-basic-smithi/5647472', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzS2w1uPBgzzeaXu3fzhtaXK3DMK7Wk+CE/+SbQ+Zg+Z4IeFqAF0GcZbtL0Z1laJqOZyOWyFJ5MCOCo2gSFzhfLqjSu3VtcJNZXQXB+8Kyd3elbgJPTolZ52s4OBP6FR4krc5P0GQhEWiSG7uAcwgl1QIFI9UiY8zvieNdo/zRIpIlpQdTEo1eNmzRmV/GbDj1l55dtLLT4yjxPX8a8yuvY+BfLEVUJAK52kkX5+tqGiLlQWKWf+DF35p8GopTbdLk/HwX+mpn3EIl22R+XCwjHFyUwXyCT9mwW3Et5A3QCDH/75pCdyx01PwRguj0lDNYlPxTxUVGij+snjsHMdEOetpD+VMKGEmKHvxkPeShxEEdX0FrX5RuEaUfZw/Om7AIImOsODO8yiP83GC4JxCaYuFZjxtqwhW7sEgJYANGWJqVPWVCpSy3kF4XDKqpeYzLkCQEoU/Ccv+98qYS3VqRSHghAbPgAb+++eVfvatsOo36V0DM52KG0xZGAua0Nb0=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi019.front.sepia.ceph.com'} 2020-11-23T13:52:09.492 INFO:teuthology.orchestra.console:Power off smithi019 2020-11-23T13:52:09.493 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-23T13:52:09.506 INFO:teuthology.orchestra.console:Power off smithi025 2020-11-23T13:52:09.507 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi025.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-23T13:52:09.515 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-23T13:52:09.530 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi025.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status