2021-05-23T02:09:49.938 INFO:root:teuthology version: 1.1.0-d92ec140 2021-05-23T02:09:50.043 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi032.front.sepia.ceph.com' reimaging 2021-05-23T02:09:50.044 INFO:teuthology.lock.ops:Updating [ubuntu@smithi032.front.sepia.ceph.com]: reset os type and version on server 2021-05-23T02:09:50.045 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2021-05-23T02:09:50.067 INFO:teuthology.lock.ops:Node 'ubuntu@smithi032.front.sepia.ceph.com' reimaging is complete 2021-05-23T02:09:50.067 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi153.front.sepia.ceph.com' reimaging 2021-05-23T02:09:50.068 INFO:teuthology.lock.ops:Updating [ubuntu@smithi153.front.sepia.ceph.com]: reset os type and version on server 2021-05-23T02:09:50.068 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2021-05-23T02:09:50.129 INFO:teuthology.lock.ops:Node 'ubuntu@smithi153.front.sepia.ceph.com' reimaging is complete 2021-05-23T02:09:50.291 INFO:teuthology.provision.fog.smithi032:Scheduling deploy of centos 8.3 2021-05-23T02:09:50.346 INFO:teuthology.provision.fog.smithi153:Scheduling deploy of centos 8.3 2021-05-23T02:09:50.632 INFO:teuthology.orchestra.console:Power off smithi032 2021-05-23T02:09:50.687 INFO:teuthology.orchestra.console:Power off smithi153 2021-05-23T02:09:59.002 INFO:teuthology.orchestra.console:Power off for smithi032 completed 2021-05-23T02:09:59.050 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2021-05-23T02:09:59.105 INFO:teuthology.orchestra.console:Power on smithi032 2021-05-23T02:09:59.153 INFO:teuthology.orchestra.console:Power on smithi153 2021-05-23T02:10:07.471 INFO:teuthology.orchestra.console:Power on for smithi032 completed 2021-05-23T02:10:07.573 INFO:teuthology.provision.fog.smithi032:Waiting for deploy to finish 2021-05-23T02:10:07.599 INFO:teuthology.orchestra.console:Power on for smithi153 completed 2021-05-23T02:10:07.700 INFO:teuthology.provision.fog.smithi153:Waiting for deploy to finish 2021-05-23T02:14:24.569 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-23T02:14:24.875 INFO:teuthology.provision.fog.smithi032:Node is ready 2021-05-23T02:14:24.891 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032.front.sepia.ceph.com 2021-05-23T02:14:24.945 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32 smithi032.front.sepia.ceph.com smithi032 2021-05-23T02:14:25.269 INFO:teuthology.provision.fog.smithi032:Deploy complete! 2021-05-23T02:14:54.895 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-23T02:14:55.254 INFO:teuthology.provision.fog.smithi153:Node is ready 2021-05-23T02:14:55.269 INFO:teuthology.orchestra.run.smithi153.stdout:smithi153.front.sepia.ceph.com 2021-05-23T02:14:55.323 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153 smithi153.front.sepia.ceph.com smithi153 2021-05-23T02:14:55.650 INFO:teuthology.provision.fog.smithi153:Deploy complete! 2021-05-23T02:14:55.785 INFO:teuthology.lock.ops:Checking smithi032.front.sepia.ceph.com 2021-05-23T02:14:55.786 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-23T02:14:55.804 INFO:teuthology.lock.ops:Checking smithi153.front.sepia.ceph.com 2021-05-23T02:14:55.804 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-23T02:14:55.819 INFO:teuthology.lock.ops:Updating [smithi032.front.sepia.ceph.com]: set os type and version on server 2021-05-23T02:14:55.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-23T02:14:56.117 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2021-05-23T02:14:56.172 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Linux" 2021-05-23T02:14:56.172 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="8" 2021-05-23T02:14:56.172 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2021-05-23T02:14:56.173 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2021-05-23T02:14:56.173 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="8" 2021-05-23T02:14:56.173 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el8" 2021-05-23T02:14:56.173 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-23T02:14:56.174 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2021-05-23T02:14:56.174 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-23T02:14:56.174 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2021-05-23T02:14:56.174 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-23T02:14:56.175 INFO:teuthology.orchestra.run.smithi032.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-23T02:14:56.175 INFO:teuthology.orchestra.run.smithi032.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-23T02:14:56.176 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2021-05-23T02:14:56.191 INFO:teuthology.lock.ops:Updating [smithi153.front.sepia.ceph.com]: set os type and version on server 2021-05-23T02:14:56.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-23T02:14:56.466 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2021-05-23T02:14:56.521 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Linux" 2021-05-23T02:14:56.521 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="8" 2021-05-23T02:14:56.521 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2021-05-23T02:14:56.521 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2021-05-23T02:14:56.522 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="8" 2021-05-23T02:14:56.522 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el8" 2021-05-23T02:14:56.522 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-23T02:14:56.522 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2021-05-23T02:14:56.523 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-23T02:14:56.523 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2021-05-23T02:14:56.523 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-23T02:14:56.523 INFO:teuthology.orchestra.run.smithi153.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-23T02:14:56.524 INFO:teuthology.orchestra.run.smithi153.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-23T02:14:56.524 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2021-05-23T02:14:56.604 INFO:teuthology.dispatcher.supervisor:Running job 6129240 2021-05-23T02:14:56.604 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/virtualenv/bin/teuthology -v --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2021-05-23_00:57:16-fs-wip-kefu-testing-2021-05-22-2207-distro-basic-smithi/6129240 --name pdonnell-2021-05-23_00:57:16-fs-wip-kefu-testing-2021-05-22-2207-distro-basic-smithi --description fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} -- /home/teuthworker/archive/pdonnell-2021-05-23_00:57:16-fs-wip-kefu-testing-2021-05-22-2207-distro-basic-smithi/6129240/orig.config.yaml 2021-05-23T02:14:56.608 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/pdonnell-2021-05-23_00:57:16-fs-wip-kefu-testing-2021-05-22-2207-distro-basic-smithi/6129240 2021-05-23T02:14:56.608 INFO:teuthology.dispatcher.supervisor:Job PID: 10122 2021-05-23T02:14:56.609 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-23T14:15:44.863 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-23T14:15:45.028 INFO:teuthology.kill:Killing Pids: {10122} 2021-05-23T14:15:46.040 INFO:teuthology.kill:Nuking machines: ['smithi032', 'smithi153'] 2021-05-23T14:15:46.903 INFO:teuthology.kill:2021-05-23 14:15:46,902.902 INFO:teuthology.nuke:targets: 2021-05-23T14:15:46.904 INFO:teuthology.kill: smithi032.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4dV1wL5f8EIze6L167od1yJlcOmh+Uz9mRTr3LwspDrR2W8u/PRf1x4Fanr+Ip7Cmq+bkPGa4dOENHQ5kbaFpTwfedkKaxrJ0M4/JXf0B+o8N1ZXmzlgukqCEl1fLe1fzB3vZYXFeabHmyXScL3r1Nm7xEbsBONs73LtkZUGTrn30Z97IB/7OzIca5+VIB8IH4GukweusNg9v0IyxYYEMs7IEWdGUaKgOnDBaMKovh1HGQ+TMmIFvxa5Tj7Qp7UnfgCnPuKzNrASxob9n6/jSACt63pgfJrG1UCvhXGnZrXfX7WRBcTySI6j+ib8Xfj2DkCU2WnT0v93Gsb1WM2/Gqc7OuixZ0DioSxH+SHpodSwOOcDPrBQXF92XxSGiIjVlYu2vRUCXId+qqDSdTS4AHJq1TXw3jIXD9VmBsyDoa1Z0elD9Ti8QtQdcFZAZCEIAHCVxW01cC2saT6ys/gCUD63e7p/TarJS2I5G/N6EJxrKXpjDEv0sj27qHS6ApBk= 2021-05-23T14:15:46.904 INFO:teuthology.kill: smithi153.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5+vjNQtBLyVIBgJoHYNp1iQrqYiL4bGSzr9xCcnzQ5JwwVld0wMQXpIg6YqjJG026E8Mlu5Pl3QAHQg8MJzEn7Fy7uDh2B1ffSqUusmYwHo6jkZFOgRY0ySq7nNMoYeaJc/Oaf/ZBgiapyyeIOOR8MeF4Z7f+mSEELws1C9x/5DXDU4QTVL31NzV0thuis12cayVRw/wjuTR6PZOTQm6TwHKZWPquAxOpQrsnLxtVDmZKZR6bQoCvgq3oBYUUe5VR///nL5F/DJXVnVj5EvonwHky4H9WcGRlyYh7nmmC36eMZSLGJOVF1W92ks2lOdNvwyDEM2P2HvucYFWfZhicC9dXJEJRC/CYekI66KeEiq5e4A/UFvQLqhiHwwkjs20NIjAQ5lka9DviTbEA/PdJI6iayKQRMShW0Cq79OMtF4IgzREJz2QrEr2P2k/LTfIyXep87IdsPmKH+19pzI0YlyBoVRjF2QGtq3qwZaNVCrikHTS3SsE7nWVeYqZjUPM= 2021-05-23T14:15:46.904 INFO:teuthology.kill:2021-05-23 14:15:46,903.903 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-23T14:15:46.905 INFO:teuthology.kill:2021-05-23 14:15:46,905.905 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-23T14:15:46.957 INFO:teuthology.kill:2021-05-23 14:15:46,957.957 INFO:teuthology.orchestra.console:Power off smithi153 2021-05-23T14:15:46.968 INFO:teuthology.kill:2021-05-23 14:15:46,968.968 INFO:teuthology.orchestra.console:Power off smithi032 2021-05-23T14:15:59.432 INFO:teuthology.kill:2021-05-23 14:15:59,432.432 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2021-05-23T14:15:59.437 INFO:teuthology.kill:2021-05-23 14:15:59,437.437 INFO:teuthology.orchestra.console:Power off for smithi032 completed 2021-05-23T14:15:59.652 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-05-23T14:15:59.653 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2021-05-23T14:15:59.653 INFO:teuthology.misc:Compressing logs... 2021-05-23T14:15:59.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-23T14:16:59.715 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-23T14:16:59.775 INFO:teuthology.kill:No teuthology processes running 2021-05-23T14:17:00.805 INFO:teuthology.kill:Nuking machines: ['smithi032', 'smithi153'] 2021-05-23T14:17:01.702 INFO:teuthology.kill:2021-05-23 14:17:01,702.702 INFO:teuthology.nuke:targets: 2021-05-23T14:17:01.703 INFO:teuthology.kill: smithi032.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4dV1wL5f8EIze6L167od1yJlcOmh+Uz9mRTr3LwspDrR2W8u/PRf1x4Fanr+Ip7Cmq+bkPGa4dOENHQ5kbaFpTwfedkKaxrJ0M4/JXf0B+o8N1ZXmzlgukqCEl1fLe1fzB3vZYXFeabHmyXScL3r1Nm7xEbsBONs73LtkZUGTrn30Z97IB/7OzIca5+VIB8IH4GukweusNg9v0IyxYYEMs7IEWdGUaKgOnDBaMKovh1HGQ+TMmIFvxa5Tj7Qp7UnfgCnPuKzNrASxob9n6/jSACt63pgfJrG1UCvhXGnZrXfX7WRBcTySI6j+ib8Xfj2DkCU2WnT0v93Gsb1WM2/Gqc7OuixZ0DioSxH+SHpodSwOOcDPrBQXF92XxSGiIjVlYu2vRUCXId+qqDSdTS4AHJq1TXw3jIXD9VmBsyDoa1Z0elD9Ti8QtQdcFZAZCEIAHCVxW01cC2saT6ys/gCUD63e7p/TarJS2I5G/N6EJxrKXpjDEv0sj27qHS6ApBk= 2021-05-23T14:17:01.703 INFO:teuthology.kill: smithi153.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5+vjNQtBLyVIBgJoHYNp1iQrqYiL4bGSzr9xCcnzQ5JwwVld0wMQXpIg6YqjJG026E8Mlu5Pl3QAHQg8MJzEn7Fy7uDh2B1ffSqUusmYwHo6jkZFOgRY0ySq7nNMoYeaJc/Oaf/ZBgiapyyeIOOR8MeF4Z7f+mSEELws1C9x/5DXDU4QTVL31NzV0thuis12cayVRw/wjuTR6PZOTQm6TwHKZWPquAxOpQrsnLxtVDmZKZR6bQoCvgq3oBYUUe5VR///nL5F/DJXVnVj5EvonwHky4H9WcGRlyYh7nmmC36eMZSLGJOVF1W92ks2lOdNvwyDEM2P2HvucYFWfZhicC9dXJEJRC/CYekI66KeEiq5e4A/UFvQLqhiHwwkjs20NIjAQ5lka9DviTbEA/PdJI6iayKQRMShW0Cq79OMtF4IgzREJz2QrEr2P2k/LTfIyXep87IdsPmKH+19pzI0YlyBoVRjF2QGtq3qwZaNVCrikHTS3SsE7nWVeYqZjUPM= 2021-05-23T14:17:01.722 INFO:teuthology.kill:2021-05-23 14:17:01,721.721 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-23T14:17:01.724 INFO:teuthology.kill:2021-05-23 14:17:01,724.724 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-23T14:17:01.782 INFO:teuthology.kill:2021-05-23 14:17:01,782.782 INFO:teuthology.orchestra.console:Power off smithi032 2021-05-23T14:17:01.795 INFO:teuthology.kill:2021-05-23 14:17:01,795.795 INFO:teuthology.orchestra.console:Power off smithi153 2021-05-23T14:17:01.930 INFO:teuthology.kill:2021-05-23 14:17:01,930.930 INFO:teuthology.orchestra.console:Power off for smithi032 completed 2021-05-23T14:17:01.942 INFO:teuthology.kill:2021-05-23 14:17:01,942.942 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2021-05-23T14:17:02.072 INFO:teuthology.kill:2021-05-23 14:17:02,072.072 INFO:teuthology.lock.ops:unlocked smithi032.front.sepia.ceph.com 2021-05-23T14:17:02.078 INFO:teuthology.kill:2021-05-23 14:17:02,077.077 INFO:teuthology.lock.ops:unlocked smithi153.front.sepia.ceph.com 2021-05-23T14:19:02.325 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15