2021-02-26T23:30:14.547 INFO:root:teuthology version: 1.1.0-7a21eca3 2021-02-26T23:30:14.658 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi085.front.sepia.ceph.com' reimaging 2021-02-26T23:30:14.659 INFO:teuthology.lock.ops:Updating [ubuntu@smithi085.front.sepia.ceph.com]: reset os type and version on server 2021-02-26T23:30:14.659 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2021-02-26T23:30:14.679 INFO:teuthology.lock.ops:Node 'ubuntu@smithi085.front.sepia.ceph.com' reimaging is complete 2021-02-26T23:30:14.679 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi121.front.sepia.ceph.com' reimaging 2021-02-26T23:30:14.679 INFO:teuthology.lock.ops:Updating [ubuntu@smithi121.front.sepia.ceph.com]: reset os type and version on server 2021-02-26T23:30:14.680 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2021-02-26T23:30:14.696 INFO:teuthology.lock.ops:Node 'ubuntu@smithi121.front.sepia.ceph.com' reimaging is complete 2021-02-26T23:30:14.881 INFO:teuthology.provision.fog.smithi085:Scheduling deploy of rhel 8.3 2021-02-26T23:30:14.900 INFO:teuthology.provision.fog.smithi121:Scheduling deploy of rhel 8.3 2021-02-26T23:30:15.228 INFO:teuthology.orchestra.console:Power off smithi085 2021-02-26T23:30:15.264 INFO:teuthology.orchestra.console:Power off smithi121 2021-02-26T23:30:23.653 INFO:teuthology.orchestra.console:Power off for smithi121 completed 2021-02-26T23:30:23.754 INFO:teuthology.orchestra.console:Power on smithi121 2021-02-26T23:30:27.755 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2021-02-26T23:30:27.857 INFO:teuthology.orchestra.console:Power on smithi085 2021-02-26T23:30:32.125 INFO:teuthology.orchestra.console:Power on for smithi085 completed 2021-02-26T23:30:32.133 INFO:teuthology.orchestra.console:Power on for smithi121 completed 2021-02-26T23:30:32.227 INFO:teuthology.provision.fog.smithi085:Waiting for deploy to finish 2021-02-26T23:30:32.234 INFO:teuthology.provision.fog.smithi121:Waiting for deploy to finish 2021-02-26T23:35:13.345 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-26T23:35:13.679 INFO:teuthology.provision.fog.smithi085:Node is ready 2021-02-26T23:35:13.693 INFO:teuthology.orchestra.run.smithi085.stdout:smithi085.front.sepia.ceph.com 2021-02-26T23:35:13.748 INFO:teuthology.orchestra.run.smithi085.stdout:172.21.15.85 smithi085.front.sepia.ceph.com smithi085 2021-02-26T23:35:14.097 INFO:teuthology.provision.fog.smithi085:Deploy complete! 2021-02-26T23:35:19.428 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-26T23:35:19.724 INFO:teuthology.provision.fog.smithi121:Node is ready 2021-02-26T23:35:19.739 INFO:teuthology.orchestra.run.smithi121.stdout:smithi121.front.sepia.ceph.com 2021-02-26T23:35:19.797 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121 smithi121.front.sepia.ceph.com smithi121 2021-02-26T23:35:20.138 INFO:teuthology.provision.fog.smithi121:Deploy complete! 2021-02-26T23:35:20.286 INFO:teuthology.lock.ops:Checking smithi085.front.sepia.ceph.com 2021-02-26T23:35:20.287 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-26T23:35:20.304 INFO:teuthology.lock.ops:Checking smithi121.front.sepia.ceph.com 2021-02-26T23:35:20.304 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-26T23:35:20.320 INFO:teuthology.lock.ops:Updating [smithi085.front.sepia.ceph.com]: set os type and version on server 2021-02-26T23:35:20.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-26T23:35:20.615 INFO:teuthology.orchestra.run.smithi085.stdout:x86_64 2021-02-26T23:35:20.669 INFO:teuthology.orchestra.run.smithi085.stdout:NAME="Red Hat Enterprise Linux" 2021-02-26T23:35:20.669 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION="8.3 (Ootpa)" 2021-02-26T23:35:20.669 INFO:teuthology.orchestra.run.smithi085.stdout:ID="rhel" 2021-02-26T23:35:20.669 INFO:teuthology.orchestra.run.smithi085.stdout:ID_LIKE="fedora" 2021-02-26T23:35:20.670 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_ID="8.3" 2021-02-26T23:35:20.670 INFO:teuthology.orchestra.run.smithi085.stdout:PLATFORM_ID="platform:el8" 2021-02-26T23:35:20.670 INFO:teuthology.orchestra.run.smithi085.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-02-26T23:35:20.670 INFO:teuthology.orchestra.run.smithi085.stdout:ANSI_COLOR="0;31" 2021-02-26T23:35:20.671 INFO:teuthology.orchestra.run.smithi085.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-02-26T23:35:20.671 INFO:teuthology.orchestra.run.smithi085.stdout:HOME_URL="https://www.redhat.com/" 2021-02-26T23:35:20.671 INFO:teuthology.orchestra.run.smithi085.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-26T23:35:20.671 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-26T23:35:20.672 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-02-26T23:35:20.672 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-02-26T23:35:20.672 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-26T23:35:20.672 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-02-26T23:35:20.674 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2021-02-26T23:35:20.690 INFO:teuthology.lock.ops:Updating [smithi121.front.sepia.ceph.com]: set os type and version on server 2021-02-26T23:35:20.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-26T23:35:20.966 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2021-02-26T23:35:21.020 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Red Hat Enterprise Linux" 2021-02-26T23:35:21.021 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8.3 (Ootpa)" 2021-02-26T23:35:21.021 INFO:teuthology.orchestra.run.smithi121.stdout:ID="rhel" 2021-02-26T23:35:21.021 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="fedora" 2021-02-26T23:35:21.021 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8.3" 2021-02-26T23:35:21.021 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8" 2021-02-26T23:35:21.022 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-02-26T23:35:21.022 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2021-02-26T23:35:21.022 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-02-26T23:35:21.022 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.redhat.com/" 2021-02-26T23:35:21.023 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-26T23:35:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: 2021-02-26T23:35:21.023 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-02-26T23:35:21.023 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-02-26T23:35:21.023 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-26T23:35:21.024 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-02-26T23:35:21.024 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2021-02-26T23:35:21.165 INFO:teuthology.dispatcher.supervisor:Running job 5917059 2021-02-26T23:35:21.166 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/virtualenv/bin/teuthology -v --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2021-02-26_22:19:00-rados-wip-sage-testing-2021-02-26-1412-distro-basic-smithi/5917059 --name sage-2021-02-26_22:19:00-rados-wip-sage-testing-2021-02-26-1412-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/redirect_set_object} -- /home/teuthworker/archive/sage-2021-02-26_22:19:00-rados-wip-sage-testing-2021-02-26-1412-distro-basic-smithi/5917059/orig.config.yaml 2021-02-26T23:35:21.173 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sage-2021-02-26_22:19:00-rados-wip-sage-testing-2021-02-26-1412-distro-basic-smithi/5917059 2021-02-26T23:35:21.173 INFO:teuthology.dispatcher.supervisor:Job PID: 4444 2021-02-26T23:35:21.174 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-02-27T11:36:11.886 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-02-27T11:36:12.164 INFO:teuthology.kill:Killing Pids: {4444} 2021-02-27T11:36:14.067 INFO:teuthology.kill:Nuking machines: ['smithi085', 'smithi121'] 2021-02-27T11:36:15.043 INFO:teuthology.kill:2021-02-27 11:36:15,043.043 INFO:teuthology.nuke:targets: 2021-02-27T11:36:15.044 INFO:teuthology.kill: smithi085.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjcRVmvqjOWlnrNxKRr4KArFoDMOpgVKHGbBV1u5lOP3Y50MRoO9q800WDP278X6uvpuZFJrj7AKPvMxOg35vAfARa+R45CqNhauipfbqGctAUI3kFu7RAeVh4slDTKVXjSSy53IU7fGAhdflCM+Qe3v8bYU0DHcDQipb4/Ar0xmUb8ezbLW5npGxPGtWkbBg2kbreOmIIQ8qAJyo1mA5JGpolRkIyGJ3xj1wEf6dSyi4Cw6JQTP4UY9o0z5IM8hQpn0B1Z9kiOl3zFMY+8dWfjeENSmk8rL1+bO+G8GO6nai8hEZp0KvdoKQ2Eaqzs98v35yHe+Fadyfa9mTai5+B/Tcv5yvMqvNFlytX34HifqxFUkD1HsrrFps7vlcKzcbPJzfbeGxpuKgj+q5gxIeY22/Lt8L2Lo8x7JfvXp3dT+efR0t/hHhJdQpwkuKBn2zcprqjhmGqsa8s/1pZTXEj5Z+v8Lc7zKkZDa9DeQJ+mFTuJNUaFES+knYMBUi8Rbc= 2021-02-27T11:36:15.044 INFO:teuthology.kill: smithi121.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrFo39W2PNF/iPrpP6mho1v3H8xRcPETewQkm4Y0xuQuERebLWY4AuCdUugfatMWcYiyiPNQwYrr67G6aZzZnmLt/ZxQur8YPsX5IkiBoarKfwOa4jzyG18I2n9rJDCsvnDyUtVPfIB0p0F+gpDv4xOb7KKx9jT/j3BEaahR6vWD4Y0WCKHHegwY0RHiZyXcx6TMj1HtvVlnXmyBH4EKZtfjfQEG8kFbQ/ppt8xjBo8ccp/8Q9vux/Elc6f7UL2BigITFTQ6gfaTk7TNThXv7oIxxBnmjaUZ0phope0VR5aM5rwFSpY8f6cMaxNlOUnGiU2QQhyjSCiVexxcWqvGkSiOimUNMqhGFY2r2+54LvQmxvatuzH+9evvoqgwovpwLGXIkIVv0KPIbCelGDDn/v3gXQ0BVPAfV26NmZuAd8su5ansEm4yDtYJCYC7Lo8zmrWzCYazUr0RVIk72uKfhr3eCEWF/WlH6bgSpIdEXMD9MsZvFaCkmuVy8kjx78OLk= 2021-02-27T11:36:15.045 INFO:teuthology.kill:2021-02-27 11:36:15,043.043 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-27T11:36:15.046 INFO:teuthology.kill:2021-02-27 11:36:15,046.046 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-27T11:36:15.111 INFO:teuthology.kill:2021-02-27 11:36:15,110.110 INFO:teuthology.orchestra.console:Power off smithi085 2021-02-27T11:36:15.124 INFO:teuthology.kill:2021-02-27 11:36:15,124.124 INFO:teuthology.orchestra.console:Power off smithi121 2021-02-27T11:36:27.591 INFO:teuthology.kill:2021-02-27 11:36:27,590.590 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2021-02-27T11:36:27.597 INFO:teuthology.kill:2021-02-27 11:36:27,597.597 INFO:teuthology.orchestra.console:Power off for smithi121 completed 2021-02-27T11:36:27.825 INFO:teuthology.task.internal:roles: ubuntu@smithi085.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-02-27T11:36:27.826 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-02-27T11:36:27.826 INFO:teuthology.misc:Compressing logs... 2021-02-27T11:36:27.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-27T11:36:46.274 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-02-27T11:36:46.339 INFO:teuthology.kill:No teuthology processes running 2021-02-27T11:36:47.398 INFO:teuthology.kill:Nuking machines: ['smithi085', 'smithi121'] 2021-02-27T11:36:48.319 INFO:teuthology.kill:2021-02-27 11:36:48,319.319 INFO:teuthology.nuke:targets: 2021-02-27T11:36:48.320 INFO:teuthology.kill: smithi085.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjcRVmvqjOWlnrNxKRr4KArFoDMOpgVKHGbBV1u5lOP3Y50MRoO9q800WDP278X6uvpuZFJrj7AKPvMxOg35vAfARa+R45CqNhauipfbqGctAUI3kFu7RAeVh4slDTKVXjSSy53IU7fGAhdflCM+Qe3v8bYU0DHcDQipb4/Ar0xmUb8ezbLW5npGxPGtWkbBg2kbreOmIIQ8qAJyo1mA5JGpolRkIyGJ3xj1wEf6dSyi4Cw6JQTP4UY9o0z5IM8hQpn0B1Z9kiOl3zFMY+8dWfjeENSmk8rL1+bO+G8GO6nai8hEZp0KvdoKQ2Eaqzs98v35yHe+Fadyfa9mTai5+B/Tcv5yvMqvNFlytX34HifqxFUkD1HsrrFps7vlcKzcbPJzfbeGxpuKgj+q5gxIeY22/Lt8L2Lo8x7JfvXp3dT+efR0t/hHhJdQpwkuKBn2zcprqjhmGqsa8s/1pZTXEj5Z+v8Lc7zKkZDa9DeQJ+mFTuJNUaFES+knYMBUi8Rbc= 2021-02-27T11:36:48.320 INFO:teuthology.kill: smithi121.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrFo39W2PNF/iPrpP6mho1v3H8xRcPETewQkm4Y0xuQuERebLWY4AuCdUugfatMWcYiyiPNQwYrr67G6aZzZnmLt/ZxQur8YPsX5IkiBoarKfwOa4jzyG18I2n9rJDCsvnDyUtVPfIB0p0F+gpDv4xOb7KKx9jT/j3BEaahR6vWD4Y0WCKHHegwY0RHiZyXcx6TMj1HtvVlnXmyBH4EKZtfjfQEG8kFbQ/ppt8xjBo8ccp/8Q9vux/Elc6f7UL2BigITFTQ6gfaTk7TNThXv7oIxxBnmjaUZ0phope0VR5aM5rwFSpY8f6cMaxNlOUnGiU2QQhyjSCiVexxcWqvGkSiOimUNMqhGFY2r2+54LvQmxvatuzH+9evvoqgwovpwLGXIkIVv0KPIbCelGDDn/v3gXQ0BVPAfV26NmZuAd8su5ansEm4yDtYJCYC7Lo8zmrWzCYazUr0RVIk72uKfhr3eCEWF/WlH6bgSpIdEXMD9MsZvFaCkmuVy8kjx78OLk= 2021-02-27T11:36:48.341 INFO:teuthology.kill:2021-02-27 11:36:48,340.340 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-27T11:36:48.344 INFO:teuthology.kill:2021-02-27 11:36:48,343.343 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-27T11:36:48.404 INFO:teuthology.kill:2021-02-27 11:36:48,404.404 INFO:teuthology.orchestra.console:Power off smithi121 2021-02-27T11:36:48.418 INFO:teuthology.kill:2021-02-27 11:36:48,416.416 INFO:teuthology.orchestra.console:Power off smithi085 2021-02-27T11:36:48.546 INFO:teuthology.kill:2021-02-27 11:36:48,546.546 INFO:teuthology.orchestra.console:Power off for smithi121 completed 2021-02-27T11:36:48.558 INFO:teuthology.kill:2021-02-27 11:36:48,558.558 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2021-02-27T11:36:48.689 INFO:teuthology.kill:2021-02-27 11:36:48,688.688 INFO:teuthology.lock.ops:unlocked smithi121.front.sepia.ceph.com 2021-02-27T11:36:48.698 INFO:teuthology.kill:2021-02-27 11:36:48,698.698 INFO:teuthology.lock.ops:unlocked smithi085.front.sepia.ceph.com 2021-02-27T11:38:48.997 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-02-27T11:38:49.098 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-02-27T11:38:49.098 INFO:teuthology.nuke:Checking targets against current locks 2021-02-27T11:38:49.189 INFO:teuthology.nuke:Not nuking smithi121.front.sepia.ceph.com because description doesn't match 2021-02-27T11:38:49.190 INFO:teuthology.nuke:Not nuking smithi085.front.sepia.ceph.com because description doesn't match