2021-02-26T22:17:52.979 INFO:root:teuthology version: 1.1.0-9a6b849c 2021-02-26T22:17:53.075 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi058.front.sepia.ceph.com' reimaging 2021-02-26T22:17:53.076 INFO:teuthology.lock.ops:Updating [ubuntu@smithi058.front.sepia.ceph.com]: reset os type and version on server 2021-02-26T22:17:53.077 INFO:teuthology.lock.ops:Updating smithi058.front.sepia.ceph.com on lock server 2021-02-26T22:17:53.092 INFO:teuthology.lock.ops:Node 'ubuntu@smithi058.front.sepia.ceph.com' reimaging is complete 2021-02-26T22:17:53.092 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi086.front.sepia.ceph.com' reimaging 2021-02-26T22:17:53.093 INFO:teuthology.lock.ops:Updating [ubuntu@smithi086.front.sepia.ceph.com]: reset os type and version on server 2021-02-26T22:17:53.093 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2021-02-26T22:17:53.107 INFO:teuthology.lock.ops:Node 'ubuntu@smithi086.front.sepia.ceph.com' reimaging is complete 2021-02-26T22:17:53.306 INFO:teuthology.provision.fog.smithi086:Scheduling deploy of rhel 8.3 2021-02-26T22:17:53.309 INFO:teuthology.provision.fog.smithi058:Scheduling deploy of rhel 8.3 2021-02-26T22:17:53.652 INFO:teuthology.orchestra.console:Power off smithi086 2021-02-26T22:17:53.665 INFO:teuthology.orchestra.console:Power off smithi058 2021-02-26T22:18:02.025 INFO:teuthology.orchestra.console:Power off for smithi058 completed 2021-02-26T22:18:02.127 INFO:teuthology.orchestra.console:Power on smithi058 2021-02-26T22:18:06.132 INFO:teuthology.orchestra.console:Power off for smithi086 completed 2021-02-26T22:18:06.233 INFO:teuthology.orchestra.console:Power on smithi086 2021-02-26T22:18:06.377 INFO:teuthology.orchestra.console:Power on for smithi058 completed 2021-02-26T22:18:06.479 INFO:teuthology.provision.fog.smithi058:Waiting for deploy to finish 2021-02-26T22:18:10.490 INFO:teuthology.orchestra.console:Power on for smithi086 completed 2021-02-26T22:18:10.591 INFO:teuthology.provision.fog.smithi086:Waiting for deploy to finish 2021-02-26T22:22:14.161 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-26T22:22:14.493 INFO:teuthology.provision.fog.smithi058:Node is ready 2021-02-26T22:22:14.509 INFO:teuthology.orchestra.run.smithi058.stdout:smithi058.front.sepia.ceph.com 2021-02-26T22:22:14.565 INFO:teuthology.orchestra.run.smithi058.stdout:172.21.15.58 smithi058.front.sepia.ceph.com smithi058 2021-02-26T22:22:14.896 INFO:teuthology.provision.fog.smithi058:Deploy complete! 2021-02-26T22:22:51.369 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-26T22:22:51.687 INFO:teuthology.provision.fog.smithi086:Node is ready 2021-02-26T22:22:51.702 INFO:teuthology.orchestra.run.smithi086.stdout:smithi086.front.sepia.ceph.com 2021-02-26T22:22:51.757 INFO:teuthology.orchestra.run.smithi086.stdout:172.21.15.86 smithi086.front.sepia.ceph.com smithi086 2021-02-26T22:22:52.086 INFO:teuthology.provision.fog.smithi086:Deploy complete! 2021-02-26T22:22:52.230 INFO:teuthology.lock.ops:Checking smithi058.front.sepia.ceph.com 2021-02-26T22:22:52.230 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-26T22:22:52.245 INFO:teuthology.lock.ops:Checking smithi086.front.sepia.ceph.com 2021-02-26T22:22:52.246 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-26T22:22:52.260 INFO:teuthology.lock.ops:Updating [smithi058.front.sepia.ceph.com]: set os type and version on server 2021-02-26T22:22:52.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-26T22:22:52.552 INFO:teuthology.orchestra.run.smithi058.stdout:x86_64 2021-02-26T22:22:52.607 INFO:teuthology.orchestra.run.smithi058.stdout:NAME="Red Hat Enterprise Linux" 2021-02-26T22:22:52.607 INFO:teuthology.orchestra.run.smithi058.stdout:VERSION="8.3 (Ootpa)" 2021-02-26T22:22:52.608 INFO:teuthology.orchestra.run.smithi058.stdout:ID="rhel" 2021-02-26T22:22:52.608 INFO:teuthology.orchestra.run.smithi058.stdout:ID_LIKE="fedora" 2021-02-26T22:22:52.608 INFO:teuthology.orchestra.run.smithi058.stdout:VERSION_ID="8.3" 2021-02-26T22:22:52.608 INFO:teuthology.orchestra.run.smithi058.stdout:PLATFORM_ID="platform:el8" 2021-02-26T22:22:52.609 INFO:teuthology.orchestra.run.smithi058.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-02-26T22:22:52.609 INFO:teuthology.orchestra.run.smithi058.stdout:ANSI_COLOR="0;31" 2021-02-26T22:22:52.609 INFO:teuthology.orchestra.run.smithi058.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-02-26T22:22:52.609 INFO:teuthology.orchestra.run.smithi058.stdout:HOME_URL="https://www.redhat.com/" 2021-02-26T22:22:52.609 INFO:teuthology.orchestra.run.smithi058.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-26T22:22:52.610 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-02-26T22:22:52.610 INFO:teuthology.orchestra.run.smithi058.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-02-26T22:22:52.610 INFO:teuthology.orchestra.run.smithi058.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-02-26T22:22:52.610 INFO:teuthology.orchestra.run.smithi058.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-26T22:22:52.611 INFO:teuthology.orchestra.run.smithi058.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-02-26T22:22:52.612 INFO:teuthology.lock.ops:Updating smithi058.front.sepia.ceph.com on lock server 2021-02-26T22:22:52.632 INFO:teuthology.lock.ops:Updating [smithi086.front.sepia.ceph.com]: set os type and version on server 2021-02-26T22:22:52.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-26T22:22:52.909 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2021-02-26T22:22:52.963 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="Red Hat Enterprise Linux" 2021-02-26T22:22:52.963 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="8.3 (Ootpa)" 2021-02-26T22:22:52.964 INFO:teuthology.orchestra.run.smithi086.stdout:ID="rhel" 2021-02-26T22:22:52.964 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="fedora" 2021-02-26T22:22:52.964 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="8.3" 2021-02-26T22:22:52.964 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el8" 2021-02-26T22:22:52.964 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-02-26T22:22:52.965 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2021-02-26T22:22:52.965 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-02-26T22:22:52.965 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://www.redhat.com/" 2021-02-26T22:22:52.965 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-26T22:22:52.965 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-02-26T22:22:52.966 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-02-26T22:22:52.966 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-02-26T22:22:52.966 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-26T22:22:52.966 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-02-26T22:22:52.968 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2021-02-26T22:22:53.037 INFO:teuthology.dispatcher.supervisor:Running job 5916942 2021-02-26T22:22:53.037 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/virtualenv/bin/teuthology -v --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2021-02-26_22:14:10-rados:cephadm:thrash-wip-mgfritch-testing-2021-02-17-1029-distro-basic-smithi/5916942 --name sage-2021-02-26_22:14:10-rados:cephadm:thrash-wip-mgfritch-testing-2021-02-17-1029-distro-basic-smithi --description rados:cephadm:thrash/{0-distro/rhel_latest 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} -- /home/teuthworker/archive/sage-2021-02-26_22:14:10-rados:cephadm:thrash-wip-mgfritch-testing-2021-02-17-1029-distro-basic-smithi/5916942/orig.config.yaml 2021-02-26T22:22:53.040 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sage-2021-02-26_22:14:10-rados:cephadm:thrash-wip-mgfritch-testing-2021-02-17-1029-distro-basic-smithi/5916942 2021-02-26T22:22:53.041 INFO:teuthology.dispatcher.supervisor:Job PID: 22573 2021-02-26T22:22:53.042 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-02-27T10:23:09.815 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-02-27T10:23:15.344 INFO:teuthology.kill:Killing Pids: {22573} 2021-02-27T10:23:16.233 INFO:teuthology.kill:Nuking machines: ['smithi058', 'smithi086'] 2021-02-27T10:23:17.012 INFO:teuthology.kill:2021-02-27 10:23:17,012.012 INFO:teuthology.nuke:targets: 2021-02-27T10:23:17.013 INFO:teuthology.kill: smithi058.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8iQUmaKrnlj+3JkRxsowXHyMTbG3M58qrl/5f64p2au9d66n+kW/nuP4I0n+aIbdNbTyqT4wCLjfuD28R+TWAuW1KlRm6bgJ/zWQ+VPdQ+egCOYCIgo4NSr3GhG2NoCnl4atAucCpiYtDfW+cucfJA9GIM9i5Of1XufsFONTGVdZMYLVJ1qEZg5Q0r0j+eamLJGDgWcPz0I9TV67GinW7/boBPJArCVmeAGdIFXIKEaVLqp+7wBDbr1wagrSMay5w9KUL82G/Oqx7m5TGcqR0mfb7/kAZ33lR361mFwVycW6ekzCi/RSAcYd4y/dW8vVKRRa8CPf925X+t5brbZA2Jjg7M2GBKpOTpvhFbTdqaJSYs62lrr9M2FaQowePTLo6I+rW+1VEvUv/oOcp+++5sCvQSZxcTOyHBf+cLb76EQCQPj3BuiUbmIlGRpFTFYts2u+O2anuXhILjsJsAOuhVYRhXsa1lI7RmUEQZoe/JCsu3WxRAv65qYoAI7QCwlc= 2021-02-27T10:23:17.013 INFO:teuthology.kill: smithi086.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklauaWxoMjzfZo0iCOQQdlKgZEVetjU3HSzvRrlpqe8MC+RGLw72QxfZm5EL9vaUT8lMHcZJN3V0FoisCX05aFaqfbFxTi0Gk0+2hmhRvny3mC4zp8wXMdR9k/giHdz4b/cPGBJHMKE2jHo+PodPkzbDaSn3tId5tU/Oj6i1wyA3at84+76ZbfBdWG3om/Su/VoWP+JAc7vZvZ6eNXcW75KCh7y9tNgMkGTHfBOLpABbKqk9cksLFoid4GLVOVzHPMybo7EKGpBwwJzpwzvr3Atqf7GxLadzZCiRCMpCWhLlWP5RVWHG8hqg11NoDH0Whp0YwRC4ZbtRUKUr2B8/bfpHXPzrY4Ag/sp4/f3FnuBAtl/1UJe900AnZYvQcWbcpg/keINJaO9SV4rp0aUqO5mhowWfzpmowSP9+fJW28oo0TvhJJcKb3J7OLu27u0Ln1NwfZ1HdqB+spKLN43rbeaVidYdapXl1RbT/m5HUyA+IIfEobpLm+1+weNYQdcU= 2021-02-27T10:23:17.013 INFO:teuthology.kill:2021-02-27 10:23:17,012.012 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-27T10:23:17.019 INFO:teuthology.kill:2021-02-27 10:23:17,019.019 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-27T10:23:17.094 INFO:teuthology.kill:2021-02-27 10:23:17,094.094 INFO:teuthology.orchestra.console:Power off smithi058 2021-02-27T10:23:17.106 INFO:teuthology.kill:2021-02-27 10:23:17,106.106 INFO:teuthology.orchestra.console:Power off smithi086 2021-02-27T10:23:29.596 INFO:teuthology.kill:2021-02-27 10:23:29,595.595 INFO:teuthology.orchestra.console:Power off for smithi058 completed 2021-02-27T10:23:29.639 INFO:teuthology.kill:2021-02-27 10:23:29,639.639 INFO:teuthology.orchestra.console:Power off for smithi086 completed 2021-02-27T10:23:29.846 INFO:teuthology.task.internal:roles: ubuntu@smithi058.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.realm.zone.a', 'node-exporter.a', 'alertmanager.a'] 2021-02-27T10:23:29.847 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2021-02-27T10:23:29.847 INFO:teuthology.misc:Compressing logs... 2021-02-27T10:23:29.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-27T10:24:29.852 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-02-27T10:24:29.903 INFO:teuthology.kill:No teuthology processes running 2021-02-27T10:24:30.853 INFO:teuthology.kill:Nuking machines: ['smithi058', 'smithi086'] 2021-02-27T10:24:31.607 INFO:teuthology.kill:2021-02-27 10:24:31,607.607 INFO:teuthology.nuke:targets: 2021-02-27T10:24:31.608 INFO:teuthology.kill: smithi058.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8iQUmaKrnlj+3JkRxsowXHyMTbG3M58qrl/5f64p2au9d66n+kW/nuP4I0n+aIbdNbTyqT4wCLjfuD28R+TWAuW1KlRm6bgJ/zWQ+VPdQ+egCOYCIgo4NSr3GhG2NoCnl4atAucCpiYtDfW+cucfJA9GIM9i5Of1XufsFONTGVdZMYLVJ1qEZg5Q0r0j+eamLJGDgWcPz0I9TV67GinW7/boBPJArCVmeAGdIFXIKEaVLqp+7wBDbr1wagrSMay5w9KUL82G/Oqx7m5TGcqR0mfb7/kAZ33lR361mFwVycW6ekzCi/RSAcYd4y/dW8vVKRRa8CPf925X+t5brbZA2Jjg7M2GBKpOTpvhFbTdqaJSYs62lrr9M2FaQowePTLo6I+rW+1VEvUv/oOcp+++5sCvQSZxcTOyHBf+cLb76EQCQPj3BuiUbmIlGRpFTFYts2u+O2anuXhILjsJsAOuhVYRhXsa1lI7RmUEQZoe/JCsu3WxRAv65qYoAI7QCwlc= 2021-02-27T10:24:31.609 INFO:teuthology.kill: smithi086.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklauaWxoMjzfZo0iCOQQdlKgZEVetjU3HSzvRrlpqe8MC+RGLw72QxfZm5EL9vaUT8lMHcZJN3V0FoisCX05aFaqfbFxTi0Gk0+2hmhRvny3mC4zp8wXMdR9k/giHdz4b/cPGBJHMKE2jHo+PodPkzbDaSn3tId5tU/Oj6i1wyA3at84+76ZbfBdWG3om/Su/VoWP+JAc7vZvZ6eNXcW75KCh7y9tNgMkGTHfBOLpABbKqk9cksLFoid4GLVOVzHPMybo7EKGpBwwJzpwzvr3Atqf7GxLadzZCiRCMpCWhLlWP5RVWHG8hqg11NoDH0Whp0YwRC4ZbtRUKUr2B8/bfpHXPzrY4Ag/sp4/f3FnuBAtl/1UJe900AnZYvQcWbcpg/keINJaO9SV4rp0aUqO5mhowWfzpmowSP9+fJW28oo0TvhJJcKb3J7OLu27u0Ln1NwfZ1HdqB+spKLN43rbeaVidYdapXl1RbT/m5HUyA+IIfEobpLm+1+weNYQdcU= 2021-02-27T10:24:31.632 INFO:teuthology.kill:2021-02-27 10:24:31,631.631 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-27T10:24:31.634 INFO:teuthology.kill:2021-02-27 10:24:31,634.634 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-27T10:24:31.694 INFO:teuthology.kill:2021-02-27 10:24:31,693.693 INFO:teuthology.orchestra.console:Power off smithi086 2021-02-27T10:24:31.710 INFO:teuthology.kill:2021-02-27 10:24:31,710.710 INFO:teuthology.orchestra.console:Power off smithi058 2021-02-27T10:24:31.844 INFO:teuthology.kill:2021-02-27 10:24:31,843.843 INFO:teuthology.orchestra.console:Power off for smithi086 completed 2021-02-27T10:24:31.854 INFO:teuthology.kill:2021-02-27 10:24:31,854.854 INFO:teuthology.orchestra.console:Power off for smithi058 completed 2021-02-27T10:24:31.994 INFO:teuthology.kill:2021-02-27 10:24:31,993.993 INFO:teuthology.lock.ops:unlocked smithi086.front.sepia.ceph.com 2021-02-27T10:24:32.000 INFO:teuthology.kill:2021-02-27 10:24:31,999.999 INFO:teuthology.lock.ops:unlocked smithi058.front.sepia.ceph.com 2021-02-27T10:26:32.190 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-02-27T10:26:32.268 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-02-27T10:26:32.268 INFO:teuthology.nuke:Checking targets against current locks 2021-02-27T10:26:32.335 INFO:teuthology.nuke:Not nuking smithi058.front.sepia.ceph.com because description doesn't match 2021-02-27T10:26:32.335 INFO:teuthology.nuke:Not nuking smithi086.front.sepia.ceph.com because description doesn't match