2021-02-11T09:54:33.451 INFO:root:teuthology version: 1.1.0-14fad317 2021-02-11T09:54:33.564 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi138.front.sepia.ceph.com' reimaging 2021-02-11T09:54:33.565 INFO:teuthology.lock.ops:Updating [ubuntu@smithi138.front.sepia.ceph.com]: reset os type and version on server 2021-02-11T09:54:33.565 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2021-02-11T09:54:33.579 INFO:teuthology.lock.ops:Node 'ubuntu@smithi138.front.sepia.ceph.com' reimaging is complete 2021-02-11T09:54:33.579 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi204.front.sepia.ceph.com' reimaging 2021-02-11T09:54:33.579 INFO:teuthology.lock.ops:Updating [ubuntu@smithi204.front.sepia.ceph.com]: reset os type and version on server 2021-02-11T09:54:33.579 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2021-02-11T09:54:33.592 INFO:teuthology.lock.ops:Node 'ubuntu@smithi204.front.sepia.ceph.com' reimaging is complete 2021-02-11T09:54:33.788 INFO:teuthology.provision.fog.smithi204:Scheduling deploy of rhel 8.3 2021-02-11T09:54:33.792 INFO:teuthology.provision.fog.smithi138:Scheduling deploy of rhel 8.3 2021-02-11T09:54:34.083 INFO:teuthology.orchestra.console:Power off smithi204 2021-02-11T09:54:34.096 INFO:teuthology.orchestra.console:Power off smithi138 2021-02-11T09:54:42.452 INFO:teuthology.orchestra.console:Power off for smithi204 completed 2021-02-11T09:54:42.553 INFO:teuthology.orchestra.console:Power on smithi204 2021-02-11T09:54:46.568 INFO:teuthology.orchestra.console:Power off for smithi138 completed 2021-02-11T09:54:46.670 INFO:teuthology.orchestra.console:Power on smithi138 2021-02-11T09:54:50.916 INFO:teuthology.orchestra.console:Power on for smithi204 completed 2021-02-11T09:54:51.016 INFO:teuthology.provision.fog.smithi204:Waiting for deploy to finish 2021-02-11T09:54:55.035 INFO:teuthology.orchestra.console:Power on for smithi138 completed 2021-02-11T09:54:55.135 INFO:teuthology.provision.fog.smithi138:Waiting for deploy to finish 2021-02-11T09:59:32.801 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-11T09:59:33.154 INFO:teuthology.provision.fog.smithi138:Node is ready 2021-02-11T09:59:33.168 INFO:teuthology.orchestra.run.smithi138.stdout:smithi138.front.sepia.ceph.com 2021-02-11T09:59:33.222 INFO:teuthology.orchestra.run.smithi138.stdout:172.21.15.138 smithi138.front.sepia.ceph.com smithi138 2021-02-11T09:59:33.539 INFO:teuthology.provision.fog.smithi138:Deploy complete! 2021-02-11T09:59:52.860 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-11T09:59:53.164 INFO:teuthology.provision.fog.smithi204:Node is ready 2021-02-11T09:59:53.178 INFO:teuthology.orchestra.run.smithi204.stdout:smithi204.front.sepia.ceph.com 2021-02-11T09:59:53.232 INFO:teuthology.orchestra.run.smithi204.stdout:172.21.15.204 smithi204.front.sepia.ceph.com smithi204 2021-02-11T09:59:53.560 INFO:teuthology.provision.fog.smithi204:Deploy complete! 2021-02-11T09:59:53.682 INFO:teuthology.lock.ops:Checking smithi138.front.sepia.ceph.com 2021-02-11T09:59:53.682 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-11T09:59:53.699 INFO:teuthology.lock.ops:Checking smithi204.front.sepia.ceph.com 2021-02-11T09:59:53.699 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-11T09:59:53.712 INFO:teuthology.lock.ops:Updating [smithi138.front.sepia.ceph.com]: set os type and version on server 2021-02-11T09:59:53.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T09:59:54.002 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2021-02-11T09:59:54.056 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="Red Hat Enterprise Linux" 2021-02-11T09:59:54.056 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="8.3 (Ootpa)" 2021-02-11T09:59:54.056 INFO:teuthology.orchestra.run.smithi138.stdout:ID="rhel" 2021-02-11T09:59:54.056 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="fedora" 2021-02-11T09:59:54.056 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="8.3" 2021-02-11T09:59:54.056 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el8" 2021-02-11T09:59:54.056 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-02-11T09:59:54.056 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31" 2021-02-11T09:59:54.056 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-02-11T09:59:54.056 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://www.redhat.com/" 2021-02-11T09:59:54.057 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-11T09:59:54.057 INFO:teuthology.orchestra.run.smithi138.stdout: 2021-02-11T09:59:54.057 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-02-11T09:59:54.057 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-02-11T09:59:54.057 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-11T09:59:54.057 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-02-11T09:59:54.058 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2021-02-11T09:59:54.074 INFO:teuthology.lock.ops:Updating [smithi204.front.sepia.ceph.com]: set os type and version on server 2021-02-11T09:59:54.074 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T09:59:54.349 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2021-02-11T09:59:54.403 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="Red Hat Enterprise Linux" 2021-02-11T09:59:54.403 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="8.3 (Ootpa)" 2021-02-11T09:59:54.403 INFO:teuthology.orchestra.run.smithi204.stdout:ID="rhel" 2021-02-11T09:59:54.403 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="fedora" 2021-02-11T09:59:54.403 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="8.3" 2021-02-11T09:59:54.403 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el8" 2021-02-11T09:59:54.403 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-02-11T09:59:54.403 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2021-02-11T09:59:54.403 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-02-11T09:59:54.403 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://www.redhat.com/" 2021-02-11T09:59:54.403 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-11T09:59:54.403 INFO:teuthology.orchestra.run.smithi204.stdout: 2021-02-11T09:59:54.404 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-02-11T09:59:54.404 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-02-11T09:59:54.404 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-11T09:59:54.404 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-02-11T09:59:54.404 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2021-02-11T09:59:54.483 INFO:teuthology.dispatcher.supervisor:Running job 5873817 2021-02-11T09:59:54.484 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology -v --owner scheduled_nojha@teuthology --archive /home/teuthworker/archive/nojha-2021-02-10_18:54:18-rados:-master-distro-basic-smithi/5873817 --name nojha-2021-02-10_18:54:18-rados:-master-distro-basic-smithi --description rados:/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} -- /home/teuthworker/archive/nojha-2021-02-10_18:54:18-rados:-master-distro-basic-smithi/5873817/orig.config.yaml 2021-02-11T09:59:54.488 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nojha-2021-02-10_18:54:18-rados:-master-distro-basic-smithi/5873817 2021-02-11T09:59:54.488 INFO:teuthology.dispatcher.supervisor:Job PID: 4847 2021-02-11T09:59:54.488 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-02-11T22:00:10.398 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-02-11T22:00:11.646 INFO:teuthology.kill:Killing Pids: {4847} 2021-02-11T22:00:13.606 INFO:teuthology.kill:Nuking machines: ['smithi138', 'smithi204'] 2021-02-11T22:00:15.291 INFO:teuthology.kill:2021-02-11 22:00:15,291.291 INFO:teuthology.nuke:targets: 2021-02-11T22:00:15.938 INFO:teuthology.kill: smithi138.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0C/+0WYhakF164tbQ0UYpdXcm5uE9wY8PufVIUSK1w8ZWEsYHFEvtM2jZfTpkIB2OGW5K9hxPBH5abjL9b/1UV97Vn4SU/E3YnrFCGJOwsInKJBayAHSd62XwFXfupM+WYk1GLMPekwgFkNXYzPP0+V7b0l+8IOFhJUnmhHk8nplEcl5IipL47F+8qtsxlyLW8CIiTRgdw3WEbbxH1V8SeETfiFOBYbSGyC4Qb1lQBcMUP1GT6Vim48QkZQX5wctR5rDix8fAvdERdW1z22tG1R74iRJgpy1hKeOQswip/HDjuXfI8GYyNzcCru9pA6/Syjzb6229HLwYQRx1pnJyE915V4T5zMqoGDpGPLMGRFn373XG+c1YYNNq1A6uxygFJoI4rZowHf+CwwYz68xEfuccOAoS2587Bb8EOnwEJ94s8tKc8mW6j2FQ40IJW1FuI59exPvIG3gKF2OXhEot9KBoRGqru0FhK1V5KnKvco5FKTL10LSVQtB90B1x3ws= 2021-02-11T22:00:15.938 INFO:teuthology.kill: smithi204.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkyFsZGmCvNvOFzE0X5waYmr/YxlYq8BIoi/747Wcm1D/rQunNL2GcXSdejKGtNUYBot5RLZ82TRe75vH1Bg/6FXnSsrfCd5k/mvGLHSWtqm2SBQInEA8gPDOtT4oC/IUCOlMZif5cLOwjwagEJgRTEbC6Kndh9PsUWHljF+C7qUNN048qTDEzFOU2vH2+cKvR3kHKZ4UHQHtBiEQ7s3RPPF/Qumjc2o5IQ0G/Tv7adUArQ+UWZpK4xyJzEAdmFCV3djZoR2VrTz6hUsCcCXYe2n1V4HDlVTOpRwcyVCjhuERBwAwWaY0WJUX/pfKk7NDEaXd4mCCnXiV1pgzcYfQ9cEkXIy4NZW+k4oDwySyjaP1gdW44cztgkKwKhEos2noQEZ9YVR7G+cULcEETQi5Dgg9S5amgeDlQhb6yM+JcgF8vjAwItNUG7qZsubZ5bx0cQMPdWuk/aw/TAhxtlGLp8959JilPaYPr/cq8zwIBVBiAom7HTf7Hzpd6KHk9W2c= 2021-02-11T22:00:15.938 INFO:teuthology.kill:2021-02-11 22:00:15,291.291 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-11T22:00:15.938 INFO:teuthology.kill:2021-02-11 22:00:15,304.304 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-11T22:00:15.938 INFO:teuthology.kill:2021-02-11 22:00:15,362.362 INFO:teuthology.orchestra.console:Power off smithi138 2021-02-11T22:00:15.938 INFO:teuthology.kill:2021-02-11 22:00:15,377.377 INFO:teuthology.orchestra.console:Power off smithi204 2021-02-11T22:00:23.774 INFO:teuthology.kill:2021-02-11 22:00:23,773.773 INFO:teuthology.orchestra.console:Power off for smithi138 completed 2021-02-11T22:00:27.879 INFO:teuthology.kill:2021-02-11 22:00:27,878.878 INFO:teuthology.orchestra.console:Power off for smithi204 completed 2021-02-11T22:00:28.145 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.realm.zone.a', 'node-exporter.a', 'alertmanager.a'] 2021-02-11T22:00:28.145 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2021-02-11T22:00:28.145 INFO:teuthology.misc:Compressing logs... 2021-02-11T22:00:28.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T22:01:28.152 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/misc.py", line 1372, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-02-11T22:01:28.227 INFO:teuthology.kill:No teuthology processes running 2021-02-11T22:01:30.150 INFO:teuthology.kill:Nuking machines: ['smithi138', 'smithi204'] 2021-02-11T22:01:31.961 INFO:teuthology.kill:2021-02-11 22:01:31,960.960 INFO:teuthology.nuke:targets: 2021-02-11T22:01:31.961 INFO:teuthology.kill: smithi138.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0C/+0WYhakF164tbQ0UYpdXcm5uE9wY8PufVIUSK1w8ZWEsYHFEvtM2jZfTpkIB2OGW5K9hxPBH5abjL9b/1UV97Vn4SU/E3YnrFCGJOwsInKJBayAHSd62XwFXfupM+WYk1GLMPekwgFkNXYzPP0+V7b0l+8IOFhJUnmhHk8nplEcl5IipL47F+8qtsxlyLW8CIiTRgdw3WEbbxH1V8SeETfiFOBYbSGyC4Qb1lQBcMUP1GT6Vim48QkZQX5wctR5rDix8fAvdERdW1z22tG1R74iRJgpy1hKeOQswip/HDjuXfI8GYyNzcCru9pA6/Syjzb6229HLwYQRx1pnJyE915V4T5zMqoGDpGPLMGRFn373XG+c1YYNNq1A6uxygFJoI4rZowHf+CwwYz68xEfuccOAoS2587Bb8EOnwEJ94s8tKc8mW6j2FQ40IJW1FuI59exPvIG3gKF2OXhEot9KBoRGqru0FhK1V5KnKvco5FKTL10LSVQtB90B1x3ws= 2021-02-11T22:01:31.962 INFO:teuthology.kill: smithi204.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkyFsZGmCvNvOFzE0X5waYmr/YxlYq8BIoi/747Wcm1D/rQunNL2GcXSdejKGtNUYBot5RLZ82TRe75vH1Bg/6FXnSsrfCd5k/mvGLHSWtqm2SBQInEA8gPDOtT4oC/IUCOlMZif5cLOwjwagEJgRTEbC6Kndh9PsUWHljF+C7qUNN048qTDEzFOU2vH2+cKvR3kHKZ4UHQHtBiEQ7s3RPPF/Qumjc2o5IQ0G/Tv7adUArQ+UWZpK4xyJzEAdmFCV3djZoR2VrTz6hUsCcCXYe2n1V4HDlVTOpRwcyVCjhuERBwAwWaY0WJUX/pfKk7NDEaXd4mCCnXiV1pgzcYfQ9cEkXIy4NZW+k4oDwySyjaP1gdW44cztgkKwKhEos2noQEZ9YVR7G+cULcEETQi5Dgg9S5amgeDlQhb6yM+JcgF8vjAwItNUG7qZsubZ5bx0cQMPdWuk/aw/TAhxtlGLp8959JilPaYPr/cq8zwIBVBiAom7HTf7Hzpd6KHk9W2c= 2021-02-11T22:01:31.989 INFO:teuthology.kill:2021-02-11 22:01:31,988.988 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-11T22:01:31.994 INFO:teuthology.kill:2021-02-11 22:01:31,994.994 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-11T22:01:32.055 INFO:teuthology.kill:2021-02-11 22:01:32,054.054 INFO:teuthology.orchestra.console:Power off smithi138 2021-02-11T22:01:32.067 INFO:teuthology.kill:2021-02-11 22:01:32,067.067 INFO:teuthology.orchestra.console:Power off smithi204 2021-02-11T22:01:32.200 INFO:teuthology.kill:2021-02-11 22:01:32,200.200 INFO:teuthology.orchestra.console:Power off for smithi138 completed 2021-02-11T22:01:32.214 INFO:teuthology.kill:2021-02-11 22:01:32,213.213 INFO:teuthology.orchestra.console:Power off for smithi204 completed 2021-02-11T22:01:32.346 INFO:teuthology.kill:2021-02-11 22:01:32,345.345 INFO:teuthology.lock.ops:unlocked smithi138.front.sepia.ceph.com 2021-02-11T22:01:32.352 INFO:teuthology.kill:2021-02-11 22:01:32,351.351 INFO:teuthology.lock.ops:unlocked smithi204.front.sepia.ceph.com 2021-02-11T22:03:32.550 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-02-11T22:03:32.631 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-02-11T22:03:32.631 INFO:teuthology.nuke:Checking targets against current locks 2021-02-11T22:03:32.710 INFO:teuthology.nuke:Not nuking smithi138.front.sepia.ceph.com because description doesn't match 2021-02-11T22:03:32.710 INFO:teuthology.nuke:Not nuking smithi204.front.sepia.ceph.com because description doesn't match