2021-05-16T20:56:46.690 INFO:root:teuthology version: 1.1.0-60f487a8 2021-05-16T20:56:46.704 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/sage-2021-05-16_20:35:35-orch:rook-wip-sage-testing-2021-05-12-1001-distro-basic-smithi/6117581 --owner scheduled_sage@teuthology --description orch:rook/smoke/{0-kubeadm 1-rook 2-workload/radosbench 3-final cluster/3-node distro/ubuntu_20.04 k8s/1.20 pod-network/flannel rook/1.6.2} --verbose --name sage-2021-05-16_20:35:35-orch:rook-wip-sage-testing-2021-05-12-1001-distro-basic-smithi -- /home/teuthworker/archive/sage-2021-05-16_20:35:35-orch:rook-wip-sage-testing-2021-05-12-1001-distro-basic-smithi/6117581/orig.config.yaml 2021-05-16T20:56:46.850 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-05-16T20:56:47.054 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sage-2021-05-16_20:35:35-orch:rook-wip-sage-testing-2021-05-12-1001-distro-basic-smithi/6117581 branch: wip-sage-testing-2021-05-12-1001 description: orch:rook/smoke/{0-kubeadm 1-rook 2-workload/radosbench 3-final cluster/3-node distro/ubuntu_20.04 k8s/1.20 pod-network/flannel rook/1.6.2} email: null first_in_suite: false job_id: '6117581' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sage-2021-05-16_20:35:35-orch:rook-wip-sage-testing-2021-05-12-1001-distro-basic-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-sage-testing-2021-05-12-1001 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: de2b467625785ff043136ca2c500d05936f331d2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: de2b467625785ff043136ca2c500d05936f331d2 kubeadm: pod_network: flannel version: '1.20' rook: rook_branch: v1.6.2 rook_image: rook/ceph:v1.6.2 workunit: branch: qa-kubeadm sha1: 60f5fb16f08a230c72f29a32b39e63bc67bdba66 owner: scheduled_sage@teuthology priority: 50 repo: git://git.ceph.com/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: de2b467625785ff043136ca2c500d05936f331d2 sleep_before_teardown: 0 suite: orch:rook suite_branch: qa-kubeadm suite_path: /home/teuthworker/src/github.com_liewegas_ceph_60f5fb16f08a230c72f29a32b39e63bc67bdba66/qa suite_relpath: qa suite_repo: https://github.com/liewegas/ceph suite_sha1: 60f5fb16f08a230c72f29a32b39e63bc67bdba66 targets: smithi116.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbCZUFOE56MIuiWpM7UiCDnhvyuQ1dJbRnrjumXiEP9aoZW+4Pn2AUTYYpDwFgjQz2aUvdvXpoOuAmCujZRuAm64eneF5zOLVYt9MFdfXe6Xk8HvL5nQ6qt9ZPprplBFLpfXSeM6SRFwbrgC6i2RH4w5gywTXmw82+kMZyPYDC52X3IFWS6PSaRY3GJ1u1+Wvj7/NP5mmZC3+wk1gEE/O5lzYqkohRy7dHMNlgS/vGZ5hPSEXYyhIIBQGhHvyTHY4YDZoTG6C5EQ45FiE0gL/I14M2CaGRy7wS9Ga/zI/K3eA1f/6+Zh2SqV01xzoObDQ0uh7tRraHXPLsqCyM/XkIpBlzBIfXpjJ34M0ZmwA+FLGw5/wN5mqFsmp6FRMbfC/tZaUuBQr0dFtD4QwUi1gwF2cmB1gcfZqBmhbUoQFtLhwmJ5wtI2g3D26EN6RmNDrUPFvM5aGspEhM/j4OsDYRYB4wYR7rzdtWZTDXt9p0K90if3t1pyiyUsYGA2fpCdk= smithi161.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDd/NYTU6Yoq9QY2SmxEmx1ptoUmWpviTThq2kVYLR5N1MHnN/OBHGEGsDV4cVejRO05z+vdQh5MUeBGq+QB8eZKpn9Yqb5+2TI+Mii8TpNaMh3pos0/K544DG8KRU6tkmg2OkEuSTVKnShtNTGWh8GKyJwdhmtY5UAbbjUBQQ2molHypDhRj7ESi3JyG8YSKlE2KJXqW/WTz8BofKx/BOBVMmcmCyB2uHzx+jFBt1Nq2wMdpHp4GC9cQYr7789uI53YP0YgYEn4cMbVNxiXpsbpiYf16YM86HNc200mg1zb6EVjmqqfByJvEdDb4LGUeN2GO+zn7RDWssZIbDF5AHWabKUTk5c8BoJKKK0ZqqoutSyJatQqC0IzvZ7mbxn8AgDf2y9/iOy4AGTsUamdbDB2DfzSj7PCUKlzHpEzE5tDe9kpxTUOMHHQzq0/ZdT7u2H/9bVIY9OIG/TA6eG947EI5+opXa9q3s6JgFiU3iekZakH8TQDLi9BRGzs3ruQI8= smithi204.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+yG5Gl8kicRwrAe9jXpNRj/U1AeFSFybQYM3U3XUIKLNZrfSvbTbxwuHy9kC97BBeDgyJnHmlnlD+enaJaGCUD5+xNmfkU0Uu2X3zP/DNgErUXCqdU7S1+y06v20rHgizngnopyHNq6+503k3ZMLa0G7MHRrjS/XojvGcrjYwlCJ2j7i54WahZzoFZsg6/zP4L3O4IkL4RLtLN0G2jL0WzAMIGkpJb5u/hhxvCcxPNaT/7gP7sqtWt4wV2d+gUTOE3SOn9emH8jueLEp6g8z2gVj0KVsSRAfqkMwKLgsPyJPdwNy5FKw/YD41G9tz84hDkZrIB7oQI5YS0r+z3nZ+azP9DVZ5MFb9H1j5qYF7jRLxchSPLH1KEs0ig7p4bFpB0vi61ZCWrZrgc/xl+txDUTvHbxUlEE2dmchP13M9FKlyIkumwXFz1b0TL3Prtb3ZM0E9kL4ZpVjZ1IE5oioaypapbVzp46NntV087sdL1Az7WAAyguGRu2Xj04Yvn/E= tasks: - kubeadm: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - install: host.a: null - radosbench: clients: - client.a - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls teuthology_branch: no-journald-to-syslog teuthology_sha1: 60f487a85454b0f19c71d4c895539c6dbdfcbd41 timestamp: 2021-05-16_20:35:35 tube: smithi user: sage verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.8274 2021-05-16T20:56:47.063 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_liewegas_ceph_60f5fb16f08a230c72f29a32b39e63bc67bdba66/qa; will attempt to use it 2021-05-16T20:56:47.086 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_liewegas_ceph_60f5fb16f08a230c72f29a32b39e63bc67bdba66/qa/tasks 2021-05-16T20:56:47.091 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-05-16T20:56:47.108 INFO:teuthology.task.internal:Checking packages... 2021-05-16T20:56:47.211 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'de2b467625785ff043136ca2c500d05936f331d2' 2021-05-16T20:56:47.212 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-05-16T20:56:47.213 INFO:teuthology.packaging:ref: None 2021-05-16T20:56:47.213 INFO:teuthology.packaging:tag: None 2021-05-16T20:56:47.213 INFO:teuthology.packaging:branch: wip-sage-testing-2021-05-12-1001 2021-05-16T20:56:47.213 INFO:teuthology.packaging:sha1: de2b467625785ff043136ca2c500d05936f331d2 2021-05-16T20:56:47.214 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-sage-testing-2021-05-12-1001 2021-05-16T20:56:47.469 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-4118-gde2b4676-1focal 2021-05-16T20:56:47.469 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-05-16T20:56:47.501 INFO:teuthology.task.internal:no buildpackages task found 2021-05-16T20:56:47.501 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-05-16T20:56:47.549 INFO:teuthology.task.internal:Saving configuration 2021-05-16T20:56:47.627 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-05-16T20:56:47.641 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-16T20:56:47.677 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-05-16 20:51:12.142317', 'description': '/home/teuthworker/archive/sage-2021-05-16_20:35:35-orch:rook-wip-sage-testing-2021-05-12-1001-distro-basic-smithi/6117581', 'locked_by': 'scheduled_sage@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICl/uEESTkrK6VtkPgsEY+OR2RAFz9WYz4fvFlmC0bb2', 'os_version': '20.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi116.front.sepia.ceph.com'} 2021-05-16T20:56:47.721 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-05-16 20:51:12.142270', 'description': '/home/teuthworker/archive/sage-2021-05-16_20:35:35-orch:rook-wip-sage-testing-2021-05-12-1001-distro-basic-smithi/6117581', 'locked_by': 'scheduled_sage@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFwHLWscrLXt0txXjzeu7w5lqaqRp38VunctxYcedBxQ', 'os_version': '20.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi161.front.sepia.ceph.com'} 2021-05-16T20:56:47.771 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-05-16 20:51:12.142301', 'description': '/home/teuthworker/archive/sage-2021-05-16_20:35:35-orch:rook-wip-sage-testing-2021-05-12-1001-distro-basic-smithi/6117581', 'locked_by': 'scheduled_sage@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICO/J/Ywwed9LyfQViR4+kh/oHxTn2XUzRv4jSPEN+ix', 'os_version': '20.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi204.front.sepia.ceph.com'} 2021-05-16T20:56:47.771 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-05-16T20:56:47.796 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['host.a', 'client.a'] 2021-05-16T20:56:47.797 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['host.b', 'client.b'] 2021-05-16T20:56:47.797 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['host.c', 'client.c'] 2021-05-16T20:56:47.797 INFO:teuthology.run_tasks:Running task console_log... 2021-05-16T20:56:48.241 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd1c1493bf8>, signals=[15]) 2021-05-16T20:56:48.242 INFO:teuthology.run_tasks:Running task internal.connect... 2021-05-16T20:56:48.257 INFO:teuthology.task.internal:Opening connections... 2021-05-16T20:56:48.257 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2021-05-16T20:56:48.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-16T20:56:48.392 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2021-05-16T20:56:48.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-16T20:57:22.896 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/teuthology/task/internal/__init__.py", line 178, in connect rem.connect() File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/teuthology/orchestra/remote.py", line 371, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/teuthology/orchestra/connection.py", line 105, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect raise NoValidConnectionsError(errors) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.161 2021-05-16T20:57:23.344 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f928847ac6534fa883e36de43627abd6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/teuthology/task/internal/__init__.py", line 178, in connect rem.connect() File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/teuthology/orchestra/remote.py", line 371, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/teuthology/orchestra/connection.py", line 105, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect raise NoValidConnectionsError(errors) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.161 2021-05-16T20:57:23.346 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-05-16T20:57:23.407 INFO:teuthology.nuke:Checking targets against current locks 2021-05-16T20:57:23.683 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/teuthology/run.py", line 412, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/teuthology/run.py", line 269, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_60f487a85454b0f19c71d4c895539c6dbdfcbd41/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable