2022-05-10T18:23:13.511 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-10T18:23:13.678 INFO:teuthology.lock.ops:Start node 'smithi019.front.sepia.ceph.com' reimaging 2022-05-10T18:23:13.679 INFO:teuthology.lock.ops:Updating [smithi019.front.sepia.ceph.com]: reset os type and version on server 2022-05-10T18:23:13.680 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2022-05-10T18:23:13.699 INFO:teuthology.lock.ops:Node 'smithi019.front.sepia.ceph.com' reimaging is complete 2022-05-10T18:23:13.699 INFO:teuthology.lock.ops:Start node 'smithi035.front.sepia.ceph.com' reimaging 2022-05-10T18:23:13.699 INFO:teuthology.lock.ops:Updating [smithi035.front.sepia.ceph.com]: reset os type and version on server 2022-05-10T18:23:13.700 INFO:teuthology.lock.ops:Updating smithi035.front.sepia.ceph.com on lock server 2022-05-10T18:23:13.720 INFO:teuthology.lock.ops:Node 'smithi035.front.sepia.ceph.com' reimaging is complete 2022-05-10T18:23:13.720 INFO:teuthology.lock.ops:Start node 'smithi081.front.sepia.ceph.com' reimaging 2022-05-10T18:23:13.721 INFO:teuthology.lock.ops:Updating [smithi081.front.sepia.ceph.com]: reset os type and version on server 2022-05-10T18:23:13.721 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2022-05-10T18:23:13.740 INFO:teuthology.lock.ops:Node 'smithi081.front.sepia.ceph.com' reimaging is complete 2022-05-10T18:23:13.902 INFO:teuthology.provision.fog.smithi019:Scheduling deploy of centos 8.stream 2022-05-10T18:23:13.955 INFO:teuthology.provision.fog.smithi035:Scheduling deploy of centos 8.stream 2022-05-10T18:23:14.008 INFO:teuthology.provision.fog.smithi081:Scheduling deploy of centos 8.stream 2022-05-10T18:23:14.422 INFO:teuthology.orchestra.console:Power off smithi035 2022-05-10T18:23:14.465 INFO:teuthology.orchestra.console:Power off smithi019 2022-05-10T18:23:14.479 INFO:teuthology.orchestra.console:Power off smithi081 2022-05-10T18:23:22.813 INFO:teuthology.orchestra.console:Power off for smithi035 completed 2022-05-10T18:23:22.822 INFO:teuthology.orchestra.console:Power off for smithi019 completed 2022-05-10T18:23:22.837 INFO:teuthology.orchestra.console:Power off for smithi081 completed 2022-05-10T18:23:22.915 INFO:teuthology.orchestra.console:Power on smithi035 2022-05-10T18:23:22.923 INFO:teuthology.orchestra.console:Power on smithi019 2022-05-10T18:23:22.941 INFO:teuthology.orchestra.console:Power on smithi081 2022-05-10T18:23:31.286 INFO:teuthology.orchestra.console:Power on for smithi019 completed 2022-05-10T18:23:31.301 INFO:teuthology.orchestra.console:Power on for smithi081 completed 2022-05-10T18:23:31.370 INFO:teuthology.orchestra.console:Power on for smithi035 completed 2022-05-10T18:23:31.388 INFO:teuthology.provision.fog.smithi019:Waiting for deploy to finish 2022-05-10T18:23:31.402 INFO:teuthology.provision.fog.smithi081:Waiting for deploy to finish 2022-05-10T18:23:31.473 INFO:teuthology.provision.fog.smithi035:Waiting for deploy to finish 2022-05-10T18:27:13.737 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-05-10T18:27:34.066 INFO:teuthology.provision.fog.smithi035:Node is ready 2022-05-10T18:27:34.102 INFO:teuthology.orchestra.run.smithi035.stdout:smithi035.front.sepia.ceph.com 2022-05-10T18:27:34.173 INFO:teuthology.orchestra.run.smithi035.stdout:172.21.15.35 smithi035.front.sepia.ceph.com smithi035 2022-05-10T18:27:34.625 INFO:teuthology.provision.fog.smithi035:Deploy complete! 2022-05-10T18:27:39.152 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-05-10T18:27:39.446 INFO:teuthology.provision.fog.smithi019:Node is ready 2022-05-10T18:27:39.465 INFO:teuthology.orchestra.run.smithi019.stdout:smithi019.front.sepia.ceph.com 2022-05-10T18:27:39.524 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19 smithi019.front.sepia.ceph.com smithi019 2022-05-10T18:27:39.875 INFO:teuthology.provision.fog.smithi019:Deploy complete! 2022-05-10T18:28:12.442 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-05-10T18:28:12.731 INFO:teuthology.provision.fog.smithi081:Node is ready 2022-05-10T18:28:12.749 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081.front.sepia.ceph.com 2022-05-10T18:28:12.807 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81 smithi081.front.sepia.ceph.com smithi081 2022-05-10T18:28:13.153 INFO:teuthology.provision.fog.smithi081:Deploy complete! 2022-05-10T18:28:13.379 INFO:teuthology.lock.ops:Checking smithi019.front.sepia.ceph.com 2022-05-10T18:28:13.380 INFO:teuthology.lock.ops:New key found. Updating... 2022-05-10T18:28:13.405 INFO:teuthology.lock.ops:Checking smithi035.front.sepia.ceph.com 2022-05-10T18:28:13.405 INFO:teuthology.lock.ops:New key found. Updating... 2022-05-10T18:28:13.424 INFO:teuthology.lock.ops:Checking smithi081.front.sepia.ceph.com 2022-05-10T18:28:13.424 INFO:teuthology.lock.ops:New key found. Updating... 2022-05-10T18:28:13.444 INFO:teuthology.lock.ops:Updating [smithi019.front.sepia.ceph.com]: set os type and version on server 2022-05-10T18:28:13.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-10T18:28:13.727 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2022-05-10T18:28:13.785 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Stream" 2022-05-10T18:28:13.786 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="8" 2022-05-10T18:28:13.786 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2022-05-10T18:28:13.786 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2022-05-10T18:28:13.786 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="8" 2022-05-10T18:28:13.787 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el8" 2022-05-10T18:28:13.787 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Stream 8" 2022-05-10T18:28:13.787 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2022-05-10T18:28:13.787 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-05-10T18:28:13.787 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://centos.org/" 2022-05-10T18:28:13.788 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-05-10T18:28:13.788 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-05-10T18:28:13.788 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-05-10T18:28:13.789 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2022-05-10T18:28:13.807 INFO:teuthology.lock.ops:Updating [smithi035.front.sepia.ceph.com]: set os type and version on server 2022-05-10T18:28:13.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi035.front.sepia.ceph.com' 2022-05-10T18:28:14.084 INFO:teuthology.orchestra.run.smithi035.stdout:x86_64 2022-05-10T18:28:14.142 INFO:teuthology.orchestra.run.smithi035.stdout:NAME="CentOS Stream" 2022-05-10T18:28:14.142 INFO:teuthology.orchestra.run.smithi035.stdout:VERSION="8" 2022-05-10T18:28:14.143 INFO:teuthology.orchestra.run.smithi035.stdout:ID="centos" 2022-05-10T18:28:14.143 INFO:teuthology.orchestra.run.smithi035.stdout:ID_LIKE="rhel fedora" 2022-05-10T18:28:14.143 INFO:teuthology.orchestra.run.smithi035.stdout:VERSION_ID="8" 2022-05-10T18:28:14.143 INFO:teuthology.orchestra.run.smithi035.stdout:PLATFORM_ID="platform:el8" 2022-05-10T18:28:14.143 INFO:teuthology.orchestra.run.smithi035.stdout:PRETTY_NAME="CentOS Stream 8" 2022-05-10T18:28:14.144 INFO:teuthology.orchestra.run.smithi035.stdout:ANSI_COLOR="0;31" 2022-05-10T18:28:14.144 INFO:teuthology.orchestra.run.smithi035.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-05-10T18:28:14.144 INFO:teuthology.orchestra.run.smithi035.stdout:HOME_URL="https://centos.org/" 2022-05-10T18:28:14.144 INFO:teuthology.orchestra.run.smithi035.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-05-10T18:28:14.145 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-05-10T18:28:14.145 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-05-10T18:28:14.145 INFO:teuthology.lock.ops:Updating smithi035.front.sepia.ceph.com on lock server 2022-05-10T18:28:14.173 INFO:teuthology.lock.ops:Updating [smithi081.front.sepia.ceph.com]: set os type and version on server 2022-05-10T18:28:14.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2022-05-10T18:28:14.386 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2022-05-10T18:28:14.443 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Stream" 2022-05-10T18:28:14.443 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="8" 2022-05-10T18:28:14.443 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos" 2022-05-10T18:28:14.444 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora" 2022-05-10T18:28:14.444 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="8" 2022-05-10T18:28:14.444 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el8" 2022-05-10T18:28:14.445 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Stream 8" 2022-05-10T18:28:14.445 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2022-05-10T18:28:14.445 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-05-10T18:28:14.445 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://centos.org/" 2022-05-10T18:28:14.446 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-05-10T18:28:14.446 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-05-10T18:28:14.446 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-05-10T18:28:14.447 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2022-05-10T18:28:14.563 INFO:teuthology.dispatcher.supervisor:Running job 6829920 2022-05-10T18:28:14.563 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-05-10_04:17:03-fs-pacific-distro-default-smithi/6829920 --name teuthology-2022-05-10_04:17:03-fs-pacific-distro-default-smithi --description fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} -- /home/teuthworker/archive/teuthology-2022-05-10_04:17:03-fs-pacific-distro-default-smithi/6829920/orig.config.yaml 2022-05-10T18:28:14.568 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-05-10_04:17:03-fs-pacific-distro-default-smithi/6829920 2022-05-10T18:28:14.569 INFO:teuthology.dispatcher.supervisor:Job PID: 452 2022-05-10T18:28:14.569 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-05-11T00:58:38.563 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T02:49:45.998 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T02:49:47.320 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T02:49:47.321 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T02:49:47.322 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T02:49:47.322 INFO:teuthology.misc:Compressing logs... 2022-05-11T02:49:47.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T02:49:47.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi035.front.sepia.ceph.com' 2022-05-11T02:49:47.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2022-05-11T02:49:55.639 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T02:49:56.823 INFO:teuthology.kill:Nuking machines: ['smithi019', 'smithi035', 'smithi081'] 2022-05-11T02:49:57.848 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-05-11T02:49:57.849 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-05-11T02:49:57.849 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated 2022-05-11T02:49:57.849 INFO:teuthology.kill: "class": algorithms.Blowfish, 2022-05-11T02:49:57.850 INFO:teuthology.kill:2022-05-11 02:49:57,848.848 INFO:teuthology.nuke:targets: 2022-05-11T02:49:57.850 INFO:teuthology.kill: smithi019.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDB28f4lisYxaD9QduWXgUYg/2pmVi+6afVjPng8kIQdKvQWLUglDL80zZyWOg08Qd6V2HDqomYZt3DoAhEr3zA3y6OBpZr9I74XToV89IalucuxHgxtIa9MTNLdH+/akidOtaX3RUdudA3hxY+EtWJEv+tS53j85QS0TMqqk1pvqdc0HiDYmvZmj1cTUmzf8h6C5xHFnUddd4ncbE9QnXUgLdotYqo4AvdcOHagXMTt0gjL5JDg8TnGXiEN4NCXh8JWrG3TsJfjve4IZW4dJcF4X1dABUyhcxOXfIL8Iz/w2JOVPck22h8bSfTd71D9oMbYx4/RBwQ9ktPK+TvhDW23eJX8MqK4TJZ2GHS131ci+sX7wpl20a6hw9Bp56XhlOwivL/oTmu0UjPAEaaBD85wIRlf1pxBOL931WTMfJswCLNS1wgUYj9eB/m9wM1luy3Ap7kGR0WQKHoLDac+0AHXf202ZeSYULpevSdXHeJQ6vi1ZZb7Su/wunaiQFX58k= 2022-05-11T02:49:57.850 INFO:teuthology.kill: smithi035.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMszTIhhhr/Fip4NJD6bTEcMvX/LIx06vrT2ULfjsReBqZ06s8W3wKEHGKgiz4ncrWnOFERkCCG1zmKp4ObOCUP7DzFCKaR5PFT/gZFcHnrXqMXaiEOk3v+nWLrK9QWNTTfMvFExJKcQmSifxL3Z0Y1pqERdniyrAu7HiwD4lAL1suGkbIqvIgl4qdKwmptJsH7KDE0OOrgQLYXQhCLOElC+MGb5lVHlxEteBezAMdg5L/qbTmIpWgJwAa8hAkdyhuacg3UvBNBeXlKqX/D/A2SOVvXgWWcwJHGqREhLg/4iVADZk3Hnv3s0ADtf1QRErvC8HqJVBQjdG76h1EjwmbOETn1ZtUsDLvZk1Dc8zCI3niaa/5MjQoFoQnJ2wi0cmp64yIynhpDTXV1z7iXZ1OXBySlwfcLgl33Sa1KmLc6Jif6pWsYbq2bKxx8Tt6aAC4RWHvTmdak2Ae0r27KojPjRC03bsA+DKklQJ6Pw6bCjpQp7dDO1pHFUrlVJuylKs= 2022-05-11T02:49:57.850 INFO:teuthology.kill: smithi081.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRZjW0zlkl1ffxp5rBndT4W9pk7hZKDudXE6pDECxGxPfLcy/nlCPYr0yxr9+0Za4Pe1956/Mu4GF8QOF9IwhJAgNOSHCyGNp9zNFANoEIxHOwE1SchK8FGe8IQSZI4cQ0zHx9j4lXqq42y5k7GPzL+KM+VbZ6GxZNza0txcNbqgn5llfb88FBE0Sz6AgtCbHd2kccV3dRtWNGGY5hGvXFRMZxQFKB2KdYXZjl//bxkE48wPJ1g6UyBUm98yE+4CmljTHCQVmAHNS2Z+IBD81ZYskJat3a1PI67XlOqiqRxQEzK645e2fuv5oTbRfQfZkz/LX5U3cUr6wBlbuRquytSL4TLfHUKBR74iv8rN1ILVd28oEa1Jn7qdLFgcAHyve6cAhfeySvRLKn0/zRjZeveN1N4FO1kuHDrjpxjI7nCkURHXZMVWmxmfc7LE6ZixAPU1dquMTVqX9B1yWVQYXcAlCqzd0xNfSsboaq1/cVDFj8DReOy8bHH0xE+KsK3f8= 2022-05-11T02:49:57.876 INFO:teuthology.kill:2022-05-11 02:49:57,876.876 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-05-11T02:49:57.879 INFO:teuthology.kill:2022-05-11 02:49:57,879.879 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-05-11T02:49:57.880 INFO:teuthology.kill:2022-05-11 02:49:57,880.880 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-05-11T02:49:57.956 INFO:teuthology.kill:2022-05-11 02:49:57,956.956 INFO:teuthology.orchestra.console:Power off smithi019 2022-05-11T02:49:57.970 INFO:teuthology.kill:2022-05-11 02:49:57,969.969 INFO:teuthology.orchestra.console:Power off smithi081 2022-05-11T02:49:57.979 INFO:teuthology.kill:2022-05-11 02:49:57,979.979 INFO:teuthology.orchestra.console:Power off smithi035 2022-05-11T02:50:06.345 INFO:teuthology.kill:2022-05-11 02:50:06,344.344 INFO:teuthology.orchestra.console:Power off for smithi035 completed 2022-05-11T02:50:06.496 INFO:teuthology.kill:2022-05-11 02:50:06,495.495 INFO:teuthology.lock.ops:unlocked: smithi035.front.sepia.ceph.com 2022-05-11T02:50:10.446 INFO:teuthology.kill:2022-05-11 02:50:10,446.446 INFO:teuthology.orchestra.console:Power off for smithi019 completed 2022-05-11T02:50:10.452 INFO:teuthology.kill:2022-05-11 02:50:10,451.451 INFO:teuthology.orchestra.console:Power off for smithi081 completed 2022-05-11T02:50:10.594 INFO:teuthology.kill:2022-05-11 02:50:10,594.594 INFO:teuthology.lock.ops:unlocked: smithi019.front.sepia.ceph.com 2022-05-11T02:50:10.598 INFO:teuthology.kill:2022-05-11 02:50:10,598.598 INFO:teuthology.lock.ops:unlocked: smithi081.front.sepia.ceph.com 2022-05-11T02:52:10.822 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T02:52:10.900 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T02:52:12.170 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T02:52:12.170 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T02:52:12.171 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T02:52:12.171 INFO:teuthology.misc:Compressing logs... 2022-05-11T02:52:12.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T02:52:30.539 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T02:52:30.619 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T02:52:31.795 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T02:54:31.939 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T02:54:32.015 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T02:54:33.346 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T02:54:33.347 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T02:54:33.347 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T02:54:33.347 INFO:teuthology.misc:Compressing logs... 2022-05-11T02:54:33.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T02:54:36.427 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T02:54:36.497 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T02:54:37.758 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T02:56:37.908 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T02:56:37.983 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T02:56:39.229 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T02:56:39.230 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T02:56:39.230 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T02:56:39.231 INFO:teuthology.misc:Compressing logs... 2022-05-11T02:56:39.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T02:56:42.283 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T02:56:42.358 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T02:56:43.536 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T02:58:43.682 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T02:58:43.768 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T02:58:45.015 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T02:58:45.016 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T02:58:45.016 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T02:58:45.016 INFO:teuthology.misc:Compressing logs... 2022-05-11T02:58:45.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T02:58:48.075 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T02:58:48.146 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T02:58:49.352 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:00:49.463 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:00:49.535 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:00:50.819 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:00:50.820 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:00:50.820 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:00:50.820 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:00:50.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:00:53.899 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:00:53.968 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:00:55.141 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:02:55.196 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:02:55.280 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:02:56.505 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:02:56.506 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:02:56.506 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:02:56.506 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:02:56.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:02:59.563 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:02:59.635 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:03:00.829 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:05:00.975 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:05:01.049 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:05:02.358 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:05:02.359 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:05:02.359 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:05:02.359 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:05:02.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:05:05.419 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:05:05.497 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:05:06.764 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:07:06.936 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:07:07.041 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:07:08.350 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:07:08.351 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:07:08.351 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:07:08.351 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:07:08.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:07:11.403 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:07:11.475 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:07:12.666 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:09:12.759 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:09:12.840 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:09:14.143 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:09:14.143 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:09:14.144 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:09:14.144 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:09:14.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:09:17.227 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:09:17.300 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:09:18.526 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:11:18.666 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:11:18.740 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:11:20.003 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:11:20.004 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:11:20.004 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:11:20.004 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:11:20.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:11:23.083 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:11:23.156 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:11:24.353 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:13:24.479 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:13:24.553 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:13:25.802 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:13:25.803 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:13:25.803 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:13:25.804 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:13:25.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:13:28.875 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:13:28.942 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:13:30.182 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:15:30.333 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:15:30.407 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:15:31.839 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:15:31.840 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:15:31.840 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:15:31.841 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:15:31.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:15:34.923 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:15:35.000 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:15:36.237 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:17:36.391 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:17:36.463 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:17:37.712 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:17:37.712 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:17:37.713 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:17:37.713 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:17:37.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:17:40.779 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:17:40.847 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:17:42.028 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:19:42.152 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:19:42.270 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:19:43.657 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:19:43.657 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:19:43.658 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:19:43.658 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:19:43.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:19:46.731 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:19:46.804 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:19:48.163 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:21:48.311 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:21:48.392 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:21:49.716 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:21:49.717 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:21:49.717 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:21:49.718 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:21:49.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:21:52.779 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:21:52.857 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:21:54.098 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:23:54.214 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:23:54.289 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:23:55.510 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:23:55.510 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:23:55.511 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:23:55.511 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:23:55.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:23:58.571 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:23:58.643 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:23:59.881 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:26:00.010 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:26:00.086 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:26:01.438 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:26:01.439 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:26:01.439 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:26:01.439 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:26:01.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:26:04.491 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:26:04.561 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:26:05.759 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:28:05.839 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:28:05.921 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:28:07.218 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:28:07.219 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:28:07.219 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:28:07.219 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:28:07.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:28:10.283 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:28:10.357 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:28:11.599 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:30:11.743 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:30:11.814 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:30:13.124 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:30:13.125 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:30:13.125 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:30:13.125 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:30:13.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:30:16.206 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:30:16.280 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:30:17.540 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:32:17.623 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:32:17.709 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:32:18.961 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:32:18.961 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:32:18.962 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:32:18.962 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:32:18.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:32:22.027 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:32:22.139 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:32:23.309 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:34:23.451 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:34:23.523 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:34:24.890 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:34:24.890 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:34:24.891 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:34:24.891 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:34:24.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:34:27.947 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:34:28.016 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:34:29.187 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:36:29.299 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:36:29.374 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:36:30.641 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:36:30.642 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:36:30.642 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:36:30.642 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:36:30.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:36:33.707 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:36:33.781 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:36:34.941 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:38:34.995 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:38:35.076 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:38:36.349 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:38:36.349 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:38:36.350 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:38:36.350 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:38:36.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:38:39.403 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:38:39.476 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:38:40.685 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:40:40.797 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:40:40.884 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:40:42.176 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:40:42.177 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:40:42.177 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:40:42.177 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:40:42.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:40:45.259 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:40:45.329 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:40:46.562 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:42:46.704 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:42:46.778 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:42:48.027 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:42:48.027 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:42:48.028 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:42:48.028 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:42:48.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:42:51.083 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:42:51.151 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:42:52.304 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:44:52.396 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:44:52.468 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:44:53.746 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:44:53.746 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:44:53.747 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:44:53.747 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:44:53.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:44:56.811 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:44:56.937 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:44:58.196 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:46:58.349 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:46:58.422 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:46:59.646 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:46:59.647 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:46:59.647 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:46:59.647 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:46:59.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:47:02.699 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:47:02.774 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:47:03.964 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:49:04.124 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:49:04.204 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:49:05.513 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:49:05.513 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:49:05.514 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:49:05.514 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:49:05.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:49:08.587 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:49:08.659 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:49:09.868 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:51:10.027 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:51:10.099 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:51:11.402 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:51:11.403 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:51:11.403 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:51:11.403 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:51:11.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:51:14.475 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:51:14.543 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:51:15.733 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:53:15.836 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:53:15.907 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:53:17.139 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:53:17.139 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:53:17.140 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:53:17.140 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:53:17.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:53:20.203 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:53:20.275 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:53:21.515 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:55:21.663 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:55:21.743 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:55:23.003 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:55:23.004 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:55:23.004 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:55:23.005 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:55:23.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:55:26.059 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:55:26.143 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:55:27.354 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:57:27.500 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:57:27.786 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:57:29.073 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:57:29.074 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:57:29.074 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:57:29.074 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:57:29.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:57:32.139 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:57:32.216 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:57:33.401 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T03:59:33.527 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T03:59:33.637 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:59:34.838 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T03:59:34.839 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T03:59:34.839 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T03:59:34.840 INFO:teuthology.misc:Compressing logs... 2022-05-11T03:59:34.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T03:59:37.899 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T03:59:37.968 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T03:59:39.151 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:01:39.207 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:01:39.282 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:01:40.507 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:01:40.508 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:01:40.508 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:01:40.509 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:01:40.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:01:43.563 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:01:43.638 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:01:44.837 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:03:44.955 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:03:45.032 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:03:46.235 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:03:46.236 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:03:46.236 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:03:46.236 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:03:46.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:03:49.291 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:03:49.361 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:03:50.644 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:05:50.798 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:05:50.876 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:05:52.089 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:05:52.090 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:05:52.090 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:05:52.091 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:05:52.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:05:55.147 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:05:55.229 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:05:56.392 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:07:56.519 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:07:56.595 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:07:57.873 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:07:57.874 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:07:57.874 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:07:57.875 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:07:57.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:08:00.939 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:08:01.011 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:08:02.175 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:10:02.328 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:10:02.400 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:10:03.691 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:10:03.691 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:10:03.691 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:10:03.692 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:10:03.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:10:06.763 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:10:06.832 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:10:08.084 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:12:08.197 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:12:08.270 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:12:09.551 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:12:09.551 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:12:09.552 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:12:09.552 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:12:09.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:12:12.619 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:12:12.687 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:12:13.851 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:14:13.935 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:14:14.013 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:14:15.287 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:14:15.288 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:14:15.288 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:14:15.288 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:14:15.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:14:18.347 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:14:18.424 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:14:19.613 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:16:19.765 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:16:19.844 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:16:21.076 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:16:21.077 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:16:21.077 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:16:21.077 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:16:21.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:16:24.139 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:16:24.214 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:16:25.418 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:18:25.566 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:18:25.640 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:18:26.865 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:18:26.866 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:18:26.866 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:18:26.867 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:18:26.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:18:29.931 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:18:30.003 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:18:31.210 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:20:31.335 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:20:31.407 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:20:32.626 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:20:32.626 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:20:32.627 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:20:32.627 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:20:32.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:20:35.691 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:20:35.768 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:20:36.931 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:22:37.067 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:22:37.142 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:22:38.362 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:22:38.363 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:22:38.363 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:22:38.364 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:22:38.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:22:41.423 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:22:41.493 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:22:42.692 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:24:42.746 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:24:42.816 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:24:44.129 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:24:44.129 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:24:44.130 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:24:44.130 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:24:44.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:24:47.211 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:24:47.280 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:24:48.462 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:26:48.554 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:26:48.627 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:26:49.997 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:26:49.997 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:26:49.997 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:26:49.998 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:26:49.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:26:53.067 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:26:53.137 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:26:54.425 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:28:54.520 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:28:54.593 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:28:55.916 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:28:55.917 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:28:55.917 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:28:55.917 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:28:55.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:28:58.987 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:28:59.061 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:29:00.267 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:31:00.387 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:31:00.460 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:31:01.781 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:31:01.782 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:31:01.782 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:31:01.783 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:31:01.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:31:04.843 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:31:04.919 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:31:06.090 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:33:06.231 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:33:06.311 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:33:07.584 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:33:07.585 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:33:07.585 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:33:07.585 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:33:07.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:33:10.667 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:33:10.744 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:33:11.982 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:35:12.123 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:35:12.198 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:35:13.484 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:35:13.484 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:35:13.485 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:35:13.485 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:35:13.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:35:16.555 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:35:16.627 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:35:17.915 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:37:18.054 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:37:18.125 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:37:19.494 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:37:19.494 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:37:19.495 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:37:19.495 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:37:19.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:37:22.571 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:37:22.643 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:37:23.868 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:39:24.018 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:39:24.087 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:39:25.328 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:39:25.328 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:39:25.329 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:39:25.329 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:39:25.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:39:28.395 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:39:28.467 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:39:29.699 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:41:29.811 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:41:29.882 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:41:31.181 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:41:31.181 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:41:31.182 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:41:31.182 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:41:31.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:41:34.251 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:41:34.323 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:41:35.549 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:43:35.680 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:43:35.755 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:43:37.040 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:43:37.041 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:43:37.041 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:43:37.041 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:43:37.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:43:40.107 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:43:40.174 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:43:41.373 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:45:41.487 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:45:41.557 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:45:42.789 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:45:42.790 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:45:42.790 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:45:42.790 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:45:42.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:45:45.867 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:45:45.939 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:45:47.110 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:47:47.267 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:47:47.337 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:47:48.590 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:47:48.590 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:47:48.591 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:47:48.591 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:47:48.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:47:51.659 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:47:51.729 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:47:52.925 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:49:53.003 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:49:53.083 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:49:54.396 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:49:54.397 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:49:54.397 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:49:54.397 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:49:54.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:49:57.451 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:49:57.523 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:49:58.676 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:51:58.826 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:51:58.895 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:52:00.143 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:52:00.144 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:52:00.144 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:52:00.144 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:52:00.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:52:03.211 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:52:03.280 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:52:04.478 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:54:04.539 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:54:04.617 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:54:05.838 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:54:05.838 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:54:05.839 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:54:05.839 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:54:05.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:54:08.907 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:54:08.978 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:54:10.130 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:56:10.305 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:56:10.377 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:56:11.671 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:56:11.672 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:56:11.672 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:56:11.673 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:56:11.673 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:56:14.731 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:56:14.799 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:56:15.981 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T04:58:16.138 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T04:58:16.217 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:58:17.571 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T04:58:17.572 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T04:58:17.572 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T04:58:17.572 INFO:teuthology.misc:Compressing logs... 2022-05-11T04:58:17.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T04:58:20.651 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T04:58:20.719 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T04:58:21.943 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:00:22.071 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T05:00:22.142 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:00:23.394 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T05:00:23.395 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T05:00:23.395 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T05:00:23.395 INFO:teuthology.misc:Compressing logs... 2022-05-11T05:00:23.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T05:00:26.475 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T05:00:26.545 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:00:27.714 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:02:27.788 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T05:02:27.861 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:02:29.217 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T05:02:29.218 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T05:02:29.218 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T05:02:29.218 INFO:teuthology.misc:Compressing logs... 2022-05-11T05:02:29.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T05:02:32.299 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T05:02:32.376 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:02:33.576 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:04:33.732 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T05:04:33.807 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:04:35.101 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T05:04:35.102 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T05:04:35.102 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T05:04:35.102 INFO:teuthology.misc:Compressing logs... 2022-05-11T05:04:35.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T05:04:38.155 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T05:04:38.229 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:04:39.439 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:06:39.590 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T05:06:39.660 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:06:40.951 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T05:06:40.951 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T05:06:40.952 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T05:06:40.952 INFO:teuthology.misc:Compressing logs... 2022-05-11T05:06:40.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T05:06:44.011 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T05:06:44.079 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:06:45.299 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:08:45.352 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T05:08:45.427 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:08:46.691 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T05:08:46.692 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T05:08:46.692 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T05:08:46.692 INFO:teuthology.misc:Compressing logs... 2022-05-11T05:08:46.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T05:08:49.771 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T05:08:49.842 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:08:51.021 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:10:51.170 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T05:10:51.247 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:10:52.556 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T05:10:52.557 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T05:10:52.557 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T05:10:52.558 INFO:teuthology.misc:Compressing logs... 2022-05-11T05:10:52.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T05:10:55.627 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T05:10:55.698 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:10:56.844 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:12:56.991 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T05:12:57.068 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:12:58.295 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T05:12:58.295 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T05:12:58.296 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T05:12:58.296 INFO:teuthology.misc:Compressing logs... 2022-05-11T05:12:58.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T05:13:01.355 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T05:13:01.424 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:13:02.624 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:15:02.772 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T05:15:02.849 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:15:04.206 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T05:15:04.206 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T05:15:04.206 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T05:15:04.207 INFO:teuthology.misc:Compressing logs... 2022-05-11T05:15:04.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T05:15:07.275 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T05:15:07.342 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:15:08.531 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:17:08.690 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T05:17:08.773 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:17:10.093 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T05:17:10.094 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T05:17:10.094 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T05:17:10.095 INFO:teuthology.misc:Compressing logs... 2022-05-11T05:17:10.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T05:17:13.163 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T05:17:13.232 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:17:14.401 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:19:14.519 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T05:19:14.591 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:19:15.853 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T05:19:15.854 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T05:19:15.854 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T05:19:15.854 INFO:teuthology.misc:Compressing logs... 2022-05-11T05:19:15.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T05:19:18.923 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T05:19:18.995 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:19:20.244 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:21:20.395 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T05:21:20.473 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:21:21.697 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T05:21:21.698 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T05:21:21.698 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T05:21:21.699 INFO:teuthology.misc:Compressing logs... 2022-05-11T05:21:21.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T05:21:24.779 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T05:21:24.849 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:21:26.029 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:23:26.133 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T05:23:26.206 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:23:27.525 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T05:23:27.525 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T05:23:27.526 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T05:23:27.526 INFO:teuthology.misc:Compressing logs... 2022-05-11T05:23:27.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T05:23:30.603 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T05:23:30.680 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:23:31.916 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:25:32.019 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T05:25:32.096 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:25:33.334 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T05:25:33.335 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T05:25:33.335 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T05:25:33.335 INFO:teuthology.misc:Compressing logs... 2022-05-11T05:25:33.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T05:25:36.395 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T05:25:36.466 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:25:37.673 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:27:37.820 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T05:27:37.899 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:27:39.184 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T05:27:39.184 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T05:27:39.185 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T05:27:39.185 INFO:teuthology.misc:Compressing logs... 2022-05-11T05:27:39.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T05:27:42.251 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T05:27:42.326 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:27:43.461 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:29:43.601 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T05:29:43.677 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:29:44.912 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-05-11T05:29:44.913 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['client.0'] 2022-05-11T05:29:44.913 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['client.1'] 2022-05-11T05:29:44.914 INFO:teuthology.misc:Compressing logs... 2022-05-11T05:29:44.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-05-11T05:29:47.979 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi019 2022-05-11T05:29:48.047 INFO:teuthology.kill:Killing Pids: {452} 2022-05-11T05:29:49.174 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-05-11T05:31:49.410 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15