2021-03-03T02:12:22.622 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-03T02:12:22.723 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi027.front.sepia.ceph.com' reimaging 2021-03-03T02:12:22.724 INFO:teuthology.lock.ops:Updating [ubuntu@smithi027.front.sepia.ceph.com]: reset os type and version on server 2021-03-03T02:12:22.725 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2021-03-03T02:12:22.740 INFO:teuthology.lock.ops:Node 'ubuntu@smithi027.front.sepia.ceph.com' reimaging is complete 2021-03-03T02:12:22.740 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi145.front.sepia.ceph.com' reimaging 2021-03-03T02:12:22.741 INFO:teuthology.lock.ops:Updating [ubuntu@smithi145.front.sepia.ceph.com]: reset os type and version on server 2021-03-03T02:12:22.741 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2021-03-03T02:12:22.755 INFO:teuthology.lock.ops:Node 'ubuntu@smithi145.front.sepia.ceph.com' reimaging is complete 2021-03-03T02:12:22.957 INFO:teuthology.provision.fog.smithi027:Scheduling deploy of rhel 8.3 2021-03-03T02:12:22.962 INFO:teuthology.provision.fog.smithi145:Scheduling deploy of rhel 8.3 2021-03-03T02:12:23.259 INFO:teuthology.orchestra.console:Power off smithi027 2021-03-03T02:12:23.272 INFO:teuthology.orchestra.console:Power off smithi145 2021-03-03T02:12:35.741 INFO:teuthology.orchestra.console:Power off for smithi027 completed 2021-03-03T02:12:35.745 INFO:teuthology.orchestra.console:Power off for smithi145 completed 2021-03-03T02:12:35.843 INFO:teuthology.orchestra.console:Power on smithi027 2021-03-03T02:12:35.849 INFO:teuthology.orchestra.console:Power on smithi145 2021-03-03T02:12:44.212 INFO:teuthology.orchestra.console:Power on for smithi027 completed 2021-03-03T02:12:44.218 INFO:teuthology.orchestra.console:Power on for smithi145 completed 2021-03-03T02:12:44.313 INFO:teuthology.provision.fog.smithi027:Waiting for deploy to finish 2021-03-03T02:12:44.318 INFO:teuthology.provision.fog.smithi145:Waiting for deploy to finish 2021-03-03T02:16:51.962 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-03T02:16:52.266 INFO:teuthology.provision.fog.smithi027:Node is ready 2021-03-03T02:16:52.281 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027.front.sepia.ceph.com 2021-03-03T02:16:52.336 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27 smithi027.front.sepia.ceph.com smithi027 2021-03-03T02:16:52.660 INFO:teuthology.provision.fog.smithi027:Deploy complete! 2021-03-03T02:17:31.261 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-03T02:17:31.551 INFO:teuthology.provision.fog.smithi145:Node is ready 2021-03-03T02:17:31.565 INFO:teuthology.orchestra.run.smithi145.stdout:smithi145.front.sepia.ceph.com 2021-03-03T02:17:31.620 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145 smithi145.front.sepia.ceph.com smithi145 2021-03-03T02:17:31.952 INFO:teuthology.provision.fog.smithi145:Deploy complete! 2021-03-03T02:17:32.077 INFO:teuthology.lock.ops:Checking smithi027.front.sepia.ceph.com 2021-03-03T02:17:32.078 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-03T02:17:32.101 INFO:teuthology.lock.ops:Checking smithi145.front.sepia.ceph.com 2021-03-03T02:17:32.102 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-03T02:17:32.116 INFO:teuthology.lock.ops:Updating [smithi027.front.sepia.ceph.com]: set os type and version on server 2021-03-03T02:17:32.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-03T02:17:32.416 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2021-03-03T02:17:32.471 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="Red Hat Enterprise Linux" 2021-03-03T02:17:32.471 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="8.3 (Ootpa)" 2021-03-03T02:17:32.471 INFO:teuthology.orchestra.run.smithi027.stdout:ID="rhel" 2021-03-03T02:17:32.471 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="fedora" 2021-03-03T02:17:32.472 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="8.3" 2021-03-03T02:17:32.472 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el8" 2021-03-03T02:17:32.472 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-03-03T02:17:32.472 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2021-03-03T02:17:32.473 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-03-03T02:17:32.473 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://www.redhat.com/" 2021-03-03T02:17:32.473 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-03-03T02:17:32.473 INFO:teuthology.orchestra.run.smithi027.stdout: 2021-03-03T02:17:32.473 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-03-03T02:17:32.474 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-03-03T02:17:32.474 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-03-03T02:17:32.474 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-03-03T02:17:32.475 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2021-03-03T02:17:32.491 INFO:teuthology.lock.ops:Updating [smithi145.front.sepia.ceph.com]: set os type and version on server 2021-03-03T02:17:32.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-03T02:17:32.770 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2021-03-03T02:17:32.824 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="Red Hat Enterprise Linux" 2021-03-03T02:17:32.825 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="8.3 (Ootpa)" 2021-03-03T02:17:32.825 INFO:teuthology.orchestra.run.smithi145.stdout:ID="rhel" 2021-03-03T02:17:32.825 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="fedora" 2021-03-03T02:17:32.825 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="8.3" 2021-03-03T02:17:32.826 INFO:teuthology.orchestra.run.smithi145.stdout:PLATFORM_ID="platform:el8" 2021-03-03T02:17:32.826 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-03-03T02:17:32.826 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31" 2021-03-03T02:17:32.826 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-03-03T02:17:32.827 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://www.redhat.com/" 2021-03-03T02:17:32.827 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-03-03T02:17:32.827 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-03-03T02:17:32.827 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-03-03T02:17:32.828 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-03-03T02:17:32.828 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-03-03T02:17:32.828 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-03-03T02:17:32.828 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2021-03-03T02:17:32.899 INFO:teuthology.dispatcher.supervisor:Running job 5927774 2021-03-03T02:17:32.900 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology -v --owner scheduled_rzarzynski@teuthology --archive /home/teuthworker/archive/rzarzynski-2021-03-02_18:30:36-rados-master-distro-basic-smithi/5927774 --name rzarzynski-2021-03-02_18:30:36-rados-master-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/small-objects} -- /home/teuthworker/archive/rzarzynski-2021-03-02_18:30:36-rados-master-distro-basic-smithi/5927774/orig.config.yaml 2021-03-03T02:17:32.903 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rzarzynski-2021-03-02_18:30:36-rados-master-distro-basic-smithi/5927774 2021-03-03T02:17:32.904 INFO:teuthology.dispatcher.supervisor:Job PID: 26227 2021-03-03T02:17:32.904 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-03T14:18:24.515 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-03T14:18:31.568 INFO:teuthology.kill:Killing Pids: {26227} 2021-03-03T14:18:32.590 INFO:teuthology.kill:Nuking machines: ['smithi027', 'smithi145'] 2021-03-03T14:18:33.477 INFO:teuthology.kill:2021-03-03 14:18:33,476.476 INFO:teuthology.nuke:targets: 2021-03-03T14:18:33.478 INFO:teuthology.kill: smithi027.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPcwH6+iGw8qLzmRKopfTw1efm12HQJFhiceed+Zx4lbGMTCCeOo/+wUSu47bSS5o0XyXDvPd6GNIpJO1w2CYwsUq+tv3V3gJUU0Vo6HKeZtD3dxkjqJB49liU6c02kjnJ7hwjaHxY06yG2FgO3dDBE6iSaxwAGdgzFN0wbc/RBCDvt4vb8s3INDzBmKe3Tr080+NbkJu21mKP/+7uurolwQLzqkZoYov54TJVHa4e0cwnPEBjRwzj/1dPFo347saHHTEynbxUfjRMJvWaz9pmLEQgNm/MAEK8BbizrMK1ItrY/wHs3fYebX+dLWKaX3zN9OtHUQltRvjZI4OiH70FynC0JmFVaPRT//yPWG2waYrFzVPue4LA6J+kGDbIopbmKXLYx+ES4Ml0A/OEcC5f7fWMMBGBcwd9qf7meznzKWKgn8tW9cWL1rlXnqmA2iWrf6yO74n93N2Bh2lu6ZFY3HUIMUrmHzdy5vKxYXZ1eJN73Ox6PDbgYAnWEAkmeuE= 2021-03-03T14:18:33.478 INFO:teuthology.kill: smithi145.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbDlDmVk3dQg86wC+VnXGPNfRdjwWGBMFO5bR3GirkYNMZ8LUFAuMz0MZABeLlES3IASWyUOSgGQ4vhGd728Gqg8EVOyT5Kumdv1C3srSw1Sbjha1a3fIx5hnCzFAFvsHa/lt9f9ihLqcIIZ9zECryB8qk1Y6NuwImg+BZG4JUvtoPmEgLBLyTh+u4JmbHQvJy+0cSHCD12qCcXLT0gg9jrCelwm1psbP12cdVfs3sVQYCaCBrTaSU+RNhATc+XGyJuaNaOjTNwAs2+/yUyFeRusk1pBPooP2vp0GKbcbHeNptOi4Bel/chhh9bLKSbb1KUqfWXp03BzaFkF7eHBr3hcwGtVvh1kAGbj2Oo+mpfqkp0buN9Ft2YLQFuPdCOqpl/L0zyXIb2yQrCMmPPoxP+Jj1M4BwxNlym3Ce3wafFInXYwVB36YhEndTq8EvC244vQGTgrNkQwOdzaUYgjz/4UydbthCcNuQBmdNOsrNaJujf8fbNGKwWy/6Cn6nlRc= 2021-03-03T14:18:33.478 INFO:teuthology.kill:2021-03-03 14:18:33,477.477 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-03T14:18:33.479 INFO:teuthology.kill:2021-03-03 14:18:33,479.479 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-03T14:18:33.539 INFO:teuthology.kill:2021-03-03 14:18:33,539.539 INFO:teuthology.orchestra.console:Power off smithi145 2021-03-03T14:18:33.552 INFO:teuthology.kill:2021-03-03 14:18:33,552.552 INFO:teuthology.orchestra.console:Power off smithi027 2021-03-03T14:18:41.903 INFO:teuthology.kill:2021-03-03 14:18:41,903.903 INFO:teuthology.orchestra.console:Power off for smithi145 completed 2021-03-03T14:18:41.915 INFO:teuthology.kill:2021-03-03 14:18:41,915.915 INFO:teuthology.orchestra.console:Power off for smithi027 completed 2021-03-03T14:18:42.138 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-03-03T14:18:42.139 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-03-03T14:18:42.139 INFO:teuthology.misc:Compressing logs... 2021-03-03T14:18:42.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-03T14:19:42.203 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-03T14:19:42.271 INFO:teuthology.kill:No teuthology processes running 2021-03-03T14:19:43.457 INFO:teuthology.kill:Nuking machines: ['smithi027', 'smithi145'] 2021-03-03T14:19:44.475 INFO:teuthology.kill:2021-03-03 14:19:44,475.475 INFO:teuthology.nuke:targets: 2021-03-03T14:19:44.476 INFO:teuthology.kill: smithi027.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPcwH6+iGw8qLzmRKopfTw1efm12HQJFhiceed+Zx4lbGMTCCeOo/+wUSu47bSS5o0XyXDvPd6GNIpJO1w2CYwsUq+tv3V3gJUU0Vo6HKeZtD3dxkjqJB49liU6c02kjnJ7hwjaHxY06yG2FgO3dDBE6iSaxwAGdgzFN0wbc/RBCDvt4vb8s3INDzBmKe3Tr080+NbkJu21mKP/+7uurolwQLzqkZoYov54TJVHa4e0cwnPEBjRwzj/1dPFo347saHHTEynbxUfjRMJvWaz9pmLEQgNm/MAEK8BbizrMK1ItrY/wHs3fYebX+dLWKaX3zN9OtHUQltRvjZI4OiH70FynC0JmFVaPRT//yPWG2waYrFzVPue4LA6J+kGDbIopbmKXLYx+ES4Ml0A/OEcC5f7fWMMBGBcwd9qf7meznzKWKgn8tW9cWL1rlXnqmA2iWrf6yO74n93N2Bh2lu6ZFY3HUIMUrmHzdy5vKxYXZ1eJN73Ox6PDbgYAnWEAkmeuE= 2021-03-03T14:19:44.477 INFO:teuthology.kill: smithi145.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbDlDmVk3dQg86wC+VnXGPNfRdjwWGBMFO5bR3GirkYNMZ8LUFAuMz0MZABeLlES3IASWyUOSgGQ4vhGd728Gqg8EVOyT5Kumdv1C3srSw1Sbjha1a3fIx5hnCzFAFvsHa/lt9f9ihLqcIIZ9zECryB8qk1Y6NuwImg+BZG4JUvtoPmEgLBLyTh+u4JmbHQvJy+0cSHCD12qCcXLT0gg9jrCelwm1psbP12cdVfs3sVQYCaCBrTaSU+RNhATc+XGyJuaNaOjTNwAs2+/yUyFeRusk1pBPooP2vp0GKbcbHeNptOi4Bel/chhh9bLKSbb1KUqfWXp03BzaFkF7eHBr3hcwGtVvh1kAGbj2Oo+mpfqkp0buN9Ft2YLQFuPdCOqpl/L0zyXIb2yQrCMmPPoxP+Jj1M4BwxNlym3Ce3wafFInXYwVB36YhEndTq8EvC244vQGTgrNkQwOdzaUYgjz/4UydbthCcNuQBmdNOsrNaJujf8fbNGKwWy/6Cn6nlRc= 2021-03-03T14:19:44.498 INFO:teuthology.kill:2021-03-03 14:19:44,498.498 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-03T14:19:44.500 INFO:teuthology.kill:2021-03-03 14:19:44,499.499 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-03T14:19:44.566 INFO:teuthology.kill:2021-03-03 14:19:44,565.565 INFO:teuthology.orchestra.console:Power off smithi027 2021-03-03T14:19:44.584 INFO:teuthology.kill:2021-03-03 14:19:44,584.584 INFO:teuthology.orchestra.console:Power off smithi145 2021-03-03T14:19:44.724 INFO:teuthology.kill:2021-03-03 14:19:44,723.723 INFO:teuthology.orchestra.console:Power off for smithi027 completed 2021-03-03T14:19:44.735 INFO:teuthology.kill:2021-03-03 14:19:44,735.735 INFO:teuthology.orchestra.console:Power off for smithi145 completed 2021-03-03T14:19:44.871 INFO:teuthology.kill:2021-03-03 14:19:44,870.870 INFO:teuthology.lock.ops:unlocked smithi027.front.sepia.ceph.com 2021-03-03T14:19:44.876 INFO:teuthology.kill:2021-03-03 14:19:44,876.876 INFO:teuthology.lock.ops:unlocked smithi145.front.sepia.ceph.com 2021-03-03T14:21:45.227 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-03-03T14:21:45.327 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-03T14:21:45.327 INFO:teuthology.nuke:Checking targets against current locks 2021-03-03T14:21:45.407 INFO:teuthology.nuke:Not nuking smithi027.front.sepia.ceph.com because description doesn't match 2021-03-03T14:21:45.408 INFO:teuthology.nuke:Not nuking smithi145.front.sepia.ceph.com because description doesn't match