2021-05-05T21:05:37.373 INFO:root:teuthology version: 1.1.0-19220a3b 2021-05-05T21:05:37.486 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi041.front.sepia.ceph.com' reimaging 2021-05-05T21:05:37.488 INFO:teuthology.lock.ops:Updating [ubuntu@smithi041.front.sepia.ceph.com]: reset os type and version on server 2021-05-05T21:05:37.488 INFO:teuthology.lock.ops:Updating smithi041.front.sepia.ceph.com on lock server 2021-05-05T21:05:37.505 INFO:teuthology.lock.ops:Node 'ubuntu@smithi041.front.sepia.ceph.com' reimaging is complete 2021-05-05T21:05:37.506 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi195.front.sepia.ceph.com' reimaging 2021-05-05T21:05:37.506 INFO:teuthology.lock.ops:Updating [ubuntu@smithi195.front.sepia.ceph.com]: reset os type and version on server 2021-05-05T21:05:37.506 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2021-05-05T21:05:37.522 INFO:teuthology.lock.ops:Node 'ubuntu@smithi195.front.sepia.ceph.com' reimaging is complete 2021-05-05T21:05:37.704 INFO:teuthology.provision.fog.smithi041:Scheduling deploy of rhel 8.3 2021-05-05T21:05:37.725 INFO:teuthology.provision.fog.smithi195:Scheduling deploy of rhel 8.3 2021-05-05T21:05:38.050 INFO:teuthology.orchestra.console:Power off smithi041 2021-05-05T21:05:38.082 INFO:teuthology.orchestra.console:Power off smithi195 2021-05-05T21:05:46.449 INFO:teuthology.orchestra.console:Power off for smithi041 completed 2021-05-05T21:05:46.455 INFO:teuthology.orchestra.console:Power off for smithi195 completed 2021-05-05T21:05:46.551 INFO:teuthology.orchestra.console:Power on smithi041 2021-05-05T21:05:46.560 INFO:teuthology.orchestra.console:Power on smithi195 2021-05-05T21:05:54.920 INFO:teuthology.orchestra.console:Power on for smithi041 completed 2021-05-05T21:05:54.925 INFO:teuthology.orchestra.console:Power on for smithi195 completed 2021-05-05T21:05:55.022 INFO:teuthology.provision.fog.smithi041:Waiting for deploy to finish 2021-05-05T21:05:55.026 INFO:teuthology.provision.fog.smithi195:Waiting for deploy to finish 2021-05-05T21:09:37.008 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-05T21:09:52.771 INFO:teuthology.provision.fog.smithi041:Node is ready 2021-05-05T21:09:52.791 INFO:teuthology.orchestra.run.smithi041.stdout:smithi041.front.sepia.ceph.com 2021-05-05T21:09:52.849 INFO:teuthology.orchestra.run.smithi041.stdout:172.21.15.41 smithi041.front.sepia.ceph.com smithi041 2021-05-05T21:09:53.215 INFO:teuthology.provision.fog.smithi041:Deploy complete! 2021-05-05T21:10:17.629 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-05T21:10:17.923 INFO:teuthology.provision.fog.smithi195:Node is ready 2021-05-05T21:10:17.979 INFO:teuthology.orchestra.run.smithi195.stdout:smithi195.front.sepia.ceph.com 2021-05-05T21:10:18.034 INFO:teuthology.orchestra.run.smithi195.stdout:172.21.15.195 smithi195.front.sepia.ceph.com smithi195 2021-05-05T21:10:18.367 INFO:teuthology.provision.fog.smithi195:Deploy complete! 2021-05-05T21:10:18.523 INFO:teuthology.lock.ops:Checking smithi041.front.sepia.ceph.com 2021-05-05T21:10:18.524 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-05T21:10:18.541 INFO:teuthology.lock.ops:Checking smithi195.front.sepia.ceph.com 2021-05-05T21:10:18.541 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-05T21:10:18.570 INFO:teuthology.lock.ops:Updating [smithi041.front.sepia.ceph.com]: set os type and version on server 2021-05-05T21:10:18.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-05T21:10:18.870 INFO:teuthology.orchestra.run.smithi041.stdout:x86_64 2021-05-05T21:10:18.926 INFO:teuthology.orchestra.run.smithi041.stdout:NAME="Red Hat Enterprise Linux" 2021-05-05T21:10:18.927 INFO:teuthology.orchestra.run.smithi041.stdout:VERSION="8.3 (Ootpa)" 2021-05-05T21:10:18.927 INFO:teuthology.orchestra.run.smithi041.stdout:ID="rhel" 2021-05-05T21:10:18.927 INFO:teuthology.orchestra.run.smithi041.stdout:ID_LIKE="fedora" 2021-05-05T21:10:18.927 INFO:teuthology.orchestra.run.smithi041.stdout:VERSION_ID="8.3" 2021-05-05T21:10:18.928 INFO:teuthology.orchestra.run.smithi041.stdout:PLATFORM_ID="platform:el8" 2021-05-05T21:10:18.928 INFO:teuthology.orchestra.run.smithi041.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-05-05T21:10:18.928 INFO:teuthology.orchestra.run.smithi041.stdout:ANSI_COLOR="0;31" 2021-05-05T21:10:18.929 INFO:teuthology.orchestra.run.smithi041.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-05-05T21:10:18.929 INFO:teuthology.orchestra.run.smithi041.stdout:HOME_URL="https://www.redhat.com/" 2021-05-05T21:10:18.929 INFO:teuthology.orchestra.run.smithi041.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-05-05T21:10:18.929 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-05-05T21:10:18.930 INFO:teuthology.orchestra.run.smithi041.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-05-05T21:10:18.930 INFO:teuthology.orchestra.run.smithi041.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-05-05T21:10:18.930 INFO:teuthology.orchestra.run.smithi041.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-05-05T21:10:18.930 INFO:teuthology.orchestra.run.smithi041.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-05-05T21:10:18.932 INFO:teuthology.lock.ops:Updating smithi041.front.sepia.ceph.com on lock server 2021-05-05T21:10:18.949 INFO:teuthology.lock.ops:Updating [smithi195.front.sepia.ceph.com]: set os type and version on server 2021-05-05T21:10:18.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-05T21:10:19.232 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2021-05-05T21:10:19.285 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="Red Hat Enterprise Linux" 2021-05-05T21:10:19.286 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="8.3 (Ootpa)" 2021-05-05T21:10:19.286 INFO:teuthology.orchestra.run.smithi195.stdout:ID="rhel" 2021-05-05T21:10:19.287 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="fedora" 2021-05-05T21:10:19.287 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="8.3" 2021-05-05T21:10:19.287 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el8" 2021-05-05T21:10:19.287 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-05-05T21:10:19.287 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2021-05-05T21:10:19.288 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-05-05T21:10:19.288 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://www.redhat.com/" 2021-05-05T21:10:19.288 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-05-05T21:10:19.288 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-05-05T21:10:19.289 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-05-05T21:10:19.289 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-05-05T21:10:19.290 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-05-05T21:10:19.290 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-05-05T21:10:19.291 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2021-05-05T21:10:19.401 INFO:teuthology.dispatcher.supervisor:Running job 6097812 2021-05-05T21:10:19.402 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology -v --owner scheduled_ksirivad@teuthology --archive /home/teuthworker/archive/ksirivad-2021-05-05_13:55:15-rados-wip-pool-start-out-with-1-pg-distro-basic-smithi/6097812 --name ksirivad-2021-05-05_13:55:15-rados-wip-pool-start-out-with-1-pg-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps} -- /home/teuthworker/archive/ksirivad-2021-05-05_13:55:15-rados-wip-pool-start-out-with-1-pg-distro-basic-smithi/6097812/orig.config.yaml 2021-05-05T21:10:19.406 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/ksirivad-2021-05-05_13:55:15-rados-wip-pool-start-out-with-1-pg-distro-basic-smithi/6097812 2021-05-05T21:10:19.406 INFO:teuthology.dispatcher.supervisor:Job PID: 22112 2021-05-05T21:10:19.407 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-06T09:11:10.224 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-06T09:11:10.414 INFO:teuthology.kill:Killing Pids: {22112} 2021-05-06T09:11:11.578 INFO:teuthology.kill:Nuking machines: ['smithi041', 'smithi195'] 2021-05-06T09:11:12.523 INFO:teuthology.kill:2021-05-06 09:11:12,523.523 INFO:teuthology.nuke:targets: 2021-05-06T09:11:12.524 INFO:teuthology.kill: smithi041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2mKVPQ1NKnHL1C5DG3QFQhFki2NaRkGcTgumKBqu6WueyDlZ3Cpnb1UkHje0A8UzGcsEGtevJY8aRSMqmEbhTWKJB0dTQT7jxwTvjDLSwKMCGkjdxbRQLIBhEmbpuTakfNBj53BMGhPQyE18hHQYQuKkQ3J6UDvjcUdG5ZIT2oyFcyG9nhW/AwYhQRsqaWiqKM1t0XkOV7JQrSdXObGj+t8u7wwIUBYgiSXcs/LwwhvwjYWcJMTnDmkMZhUM2HPMKGh9mBkDH7CBYcN9N1yD8pWTJQXtQopOFiDxgmF/8eEaJY87WsybmoVllHDS8Jzf3uOiaGBEz+0VtODUUg5/1u1sDw6+Gy08kOhJtYSF7guRmsejULSZsBiZIDvvnmU3TjNU5Ut9LGU4ZhJ/IiHn4JX9sQVeKeJegUOyX6gl95UUwpTetfZ8WfrRhINE7pWR6Nlr4Dq9qTLb2nYIK+Ru3LCI5/pK/r5mNY6q7cOtjwHyi6sOZ0PNuWktmYsiQn+M= 2021-05-06T09:11:12.524 INFO:teuthology.kill: smithi195.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz9iwcdgp8BQlPUIfXfLe4IJowhXn2/gykn+BrixGRZSNO4I2i4UhPeRLfP5RK7lw0KxbrRvFSjoyJB4C8Vfm06uzu0ziCC++UQem7plUjTx/BoS1EeRKmn6yPP/11d637gSJLEd7xly7JD2/yocjn9VE9/aqIC8VHlzDcjSi9mogKafzci5NiTfx8qV8Hp5kh65rvAak4DLcFPJvyr2QztHKjBaNPeaw03iBakKh3UAB9I/bBEETXlV4/AoSIyxFoL7x1a+e4aIy8fwlpFK4a2opvjCtxtYHhR3BLV1O8EtvLjvZNu7d8bbAX/l+t7rqiMNq0DUQjQX38pVBl1GDK2yz2pjCWDMXr7qEYe78WfiHE00DlcvKYjYhL2dLVEPrq+3SshGNv2XZH5OL1PDEstSgPqlM/b3f4AyloOjrrPsM0wy9RdM+WZQvj3o9cavzvEI9CwIjKIwoQUqYjaq9y1MUKXgD3tz6i9eohiW9YAl1kf44jfd1QOTu59WhSs48= 2021-05-06T09:11:12.524 INFO:teuthology.kill:2021-05-06 09:11:12,523.523 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-06T09:11:12.525 INFO:teuthology.kill:2021-05-06 09:11:12,525.525 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-06T09:11:12.593 INFO:teuthology.kill:2021-05-06 09:11:12,593.593 INFO:teuthology.orchestra.console:Power off smithi041 2021-05-06T09:11:12.608 INFO:teuthology.kill:2021-05-06 09:11:12,608.608 INFO:teuthology.orchestra.console:Power off smithi195 2021-05-06T09:11:20.966 INFO:teuthology.kill:2021-05-06 09:11:20,966.966 INFO:teuthology.orchestra.console:Power off for smithi041 completed 2021-05-06T09:11:20.974 INFO:teuthology.kill:2021-05-06 09:11:20,974.974 INFO:teuthology.orchestra.console:Power off for smithi195 completed 2021-05-06T09:11:21.203 INFO:teuthology.task.internal:roles: ubuntu@smithi041.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-05-06T09:11:21.204 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-05-06T09:11:21.204 INFO:teuthology.misc:Compressing logs... 2021-05-06T09:11:21.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-06T09:11:55.740 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-06T09:11:55.815 INFO:teuthology.kill:No teuthology processes running 2021-05-06T09:11:56.897 INFO:teuthology.kill:Nuking machines: ['smithi041', 'smithi195'] 2021-05-06T09:11:57.875 INFO:teuthology.kill:2021-05-06 09:11:57,874.874 INFO:teuthology.nuke:targets: 2021-05-06T09:11:57.876 INFO:teuthology.kill: smithi041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2mKVPQ1NKnHL1C5DG3QFQhFki2NaRkGcTgumKBqu6WueyDlZ3Cpnb1UkHje0A8UzGcsEGtevJY8aRSMqmEbhTWKJB0dTQT7jxwTvjDLSwKMCGkjdxbRQLIBhEmbpuTakfNBj53BMGhPQyE18hHQYQuKkQ3J6UDvjcUdG5ZIT2oyFcyG9nhW/AwYhQRsqaWiqKM1t0XkOV7JQrSdXObGj+t8u7wwIUBYgiSXcs/LwwhvwjYWcJMTnDmkMZhUM2HPMKGh9mBkDH7CBYcN9N1yD8pWTJQXtQopOFiDxgmF/8eEaJY87WsybmoVllHDS8Jzf3uOiaGBEz+0VtODUUg5/1u1sDw6+Gy08kOhJtYSF7guRmsejULSZsBiZIDvvnmU3TjNU5Ut9LGU4ZhJ/IiHn4JX9sQVeKeJegUOyX6gl95UUwpTetfZ8WfrRhINE7pWR6Nlr4Dq9qTLb2nYIK+Ru3LCI5/pK/r5mNY6q7cOtjwHyi6sOZ0PNuWktmYsiQn+M= 2021-05-06T09:11:57.876 INFO:teuthology.kill: smithi195.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz9iwcdgp8BQlPUIfXfLe4IJowhXn2/gykn+BrixGRZSNO4I2i4UhPeRLfP5RK7lw0KxbrRvFSjoyJB4C8Vfm06uzu0ziCC++UQem7plUjTx/BoS1EeRKmn6yPP/11d637gSJLEd7xly7JD2/yocjn9VE9/aqIC8VHlzDcjSi9mogKafzci5NiTfx8qV8Hp5kh65rvAak4DLcFPJvyr2QztHKjBaNPeaw03iBakKh3UAB9I/bBEETXlV4/AoSIyxFoL7x1a+e4aIy8fwlpFK4a2opvjCtxtYHhR3BLV1O8EtvLjvZNu7d8bbAX/l+t7rqiMNq0DUQjQX38pVBl1GDK2yz2pjCWDMXr7qEYe78WfiHE00DlcvKYjYhL2dLVEPrq+3SshGNv2XZH5OL1PDEstSgPqlM/b3f4AyloOjrrPsM0wy9RdM+WZQvj3o9cavzvEI9CwIjKIwoQUqYjaq9y1MUKXgD3tz6i9eohiW9YAl1kf44jfd1QOTu59WhSs48= 2021-05-06T09:11:57.897 INFO:teuthology.kill:2021-05-06 09:11:57,897.897 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-06T09:11:57.900 INFO:teuthology.kill:2021-05-06 09:11:57,900.900 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-06T09:11:57.965 INFO:teuthology.kill:2021-05-06 09:11:57,964.964 INFO:teuthology.orchestra.console:Power off smithi195 2021-05-06T09:11:57.979 INFO:teuthology.kill:2021-05-06 09:11:57,978.978 INFO:teuthology.orchestra.console:Power off smithi041 2021-05-06T09:11:58.106 INFO:teuthology.kill:2021-05-06 09:11:58,106.106 INFO:teuthology.orchestra.console:Power off for smithi195 completed 2021-05-06T09:11:58.120 INFO:teuthology.kill:2021-05-06 09:11:58,120.120 INFO:teuthology.orchestra.console:Power off for smithi041 completed 2021-05-06T09:11:58.250 INFO:teuthology.kill:2021-05-06 09:11:58,250.250 INFO:teuthology.lock.ops:unlocked smithi195.front.sepia.ceph.com 2021-05-06T09:11:58.259 INFO:teuthology.kill:2021-05-06 09:11:58,259.259 INFO:teuthology.lock.ops:unlocked smithi041.front.sepia.ceph.com 2021-05-06T09:13:58.511 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-05-06T09:13:58.961 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-05-06T09:13:59.084 INFO:teuthology.nuke:Checking targets against current locks 2021-05-06T09:13:59.211 INFO:teuthology.nuke:Not nuking smithi195.front.sepia.ceph.com because description doesn't match 2021-05-06T09:13:59.250 INFO:teuthology.nuke:Not nuking smithi041.front.sepia.ceph.com because description doesn't match