2021-02-27T15:49:29.701 INFO:root:teuthology version: 1.1.0-7a21eca3 2021-02-27T15:49:29.808 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi075.front.sepia.ceph.com' reimaging 2021-02-27T15:49:29.809 INFO:teuthology.lock.ops:Updating [ubuntu@smithi075.front.sepia.ceph.com]: reset os type and version on server 2021-02-27T15:49:29.810 INFO:teuthology.lock.ops:Updating smithi075.front.sepia.ceph.com on lock server 2021-02-27T15:49:29.826 INFO:teuthology.lock.ops:Node 'ubuntu@smithi075.front.sepia.ceph.com' reimaging is complete 2021-02-27T15:49:29.826 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi153.front.sepia.ceph.com' reimaging 2021-02-27T15:49:29.827 INFO:teuthology.lock.ops:Updating [ubuntu@smithi153.front.sepia.ceph.com]: reset os type and version on server 2021-02-27T15:49:29.827 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2021-02-27T15:49:29.844 INFO:teuthology.lock.ops:Node 'ubuntu@smithi153.front.sepia.ceph.com' reimaging is complete 2021-02-27T15:49:30.034 INFO:teuthology.provision.fog.smithi075:Scheduling deploy of centos 8.2 2021-02-27T15:49:30.072 INFO:teuthology.provision.fog.smithi153:Scheduling deploy of centos 8.2 2021-02-27T15:49:30.407 INFO:teuthology.orchestra.console:Power off smithi075 2021-02-27T15:49:30.449 INFO:teuthology.orchestra.console:Power off smithi153 2021-02-27T15:49:30.583 INFO:teuthology.orchestra.console:Power off for smithi075 completed 2021-02-27T15:49:30.601 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2021-02-27T15:49:30.685 INFO:teuthology.orchestra.console:Power on smithi075 2021-02-27T15:49:30.702 INFO:teuthology.orchestra.console:Power on smithi153 2021-02-27T15:49:34.958 INFO:teuthology.orchestra.console:Power on for smithi153 completed 2021-02-27T15:49:35.060 INFO:teuthology.provision.fog.smithi153:Waiting for deploy to finish 2021-02-27T15:49:39.059 INFO:teuthology.orchestra.console:Power on for smithi075 completed 2021-02-27T15:49:39.160 INFO:teuthology.provision.fog.smithi075:Waiting for deploy to finish 2021-02-27T15:54:05.411 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-27T15:54:10.927 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-27T15:54:11.222 INFO:teuthology.provision.fog.smithi075:Node is ready 2021-02-27T15:54:11.236 INFO:teuthology.orchestra.run.smithi075.stdout:smithi075.front.sepia.ceph.com 2021-02-27T15:54:11.290 INFO:teuthology.orchestra.run.smithi075.stdout:172.21.15.75 smithi075.front.sepia.ceph.com smithi075 2021-02-27T15:54:11.609 INFO:teuthology.provision.fog.smithi075:Deploy complete! 2021-02-27T15:54:25.781 INFO:teuthology.provision.fog.smithi153:Node is ready 2021-02-27T15:54:25.800 INFO:teuthology.orchestra.run.smithi153.stdout:smithi153.front.sepia.ceph.com 2021-02-27T15:54:25.858 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153 smithi153.front.sepia.ceph.com smithi153 2021-02-27T15:54:26.210 INFO:teuthology.provision.fog.smithi153:Deploy complete! 2021-02-27T15:54:26.351 INFO:teuthology.lock.ops:Checking smithi075.front.sepia.ceph.com 2021-02-27T15:54:26.358 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-27T15:54:26.380 INFO:teuthology.lock.ops:Checking smithi153.front.sepia.ceph.com 2021-02-27T15:54:26.381 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-27T15:54:26.397 INFO:teuthology.lock.ops:Updating [smithi075.front.sepia.ceph.com]: set os type and version on server 2021-02-27T15:54:26.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-27T15:54:26.683 INFO:teuthology.orchestra.run.smithi075.stdout:x86_64 2021-02-27T15:54:26.737 INFO:teuthology.orchestra.run.smithi075.stdout:NAME="CentOS Linux" 2021-02-27T15:54:26.737 INFO:teuthology.orchestra.run.smithi075.stdout:VERSION="8 (Core)" 2021-02-27T15:54:26.738 INFO:teuthology.orchestra.run.smithi075.stdout:ID="centos" 2021-02-27T15:54:26.738 INFO:teuthology.orchestra.run.smithi075.stdout:ID_LIKE="rhel fedora" 2021-02-27T15:54:26.738 INFO:teuthology.orchestra.run.smithi075.stdout:VERSION_ID="8" 2021-02-27T15:54:26.738 INFO:teuthology.orchestra.run.smithi075.stdout:PLATFORM_ID="platform:el8" 2021-02-27T15:54:26.739 INFO:teuthology.orchestra.run.smithi075.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-02-27T15:54:26.739 INFO:teuthology.orchestra.run.smithi075.stdout:ANSI_COLOR="0;31" 2021-02-27T15:54:26.739 INFO:teuthology.orchestra.run.smithi075.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-02-27T15:54:26.739 INFO:teuthology.orchestra.run.smithi075.stdout:HOME_URL="https://www.centos.org/" 2021-02-27T15:54:26.740 INFO:teuthology.orchestra.run.smithi075.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-27T15:54:26.740 INFO:teuthology.orchestra.run.smithi075.stdout: 2021-02-27T15:54:26.740 INFO:teuthology.orchestra.run.smithi075.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-02-27T15:54:26.740 INFO:teuthology.orchestra.run.smithi075.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-02-27T15:54:26.741 INFO:teuthology.orchestra.run.smithi075.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-27T15:54:26.741 INFO:teuthology.orchestra.run.smithi075.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-02-27T15:54:26.741 INFO:teuthology.orchestra.run.smithi075.stdout: 2021-02-27T15:54:26.742 INFO:teuthology.lock.ops:Updating smithi075.front.sepia.ceph.com on lock server 2021-02-27T15:54:26.760 INFO:teuthology.lock.ops:Updating [smithi153.front.sepia.ceph.com]: set os type and version on server 2021-02-27T15:54:26.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-27T15:54:27.038 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2021-02-27T15:54:27.096 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Linux" 2021-02-27T15:54:27.096 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="8 (Core)" 2021-02-27T15:54:27.096 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2021-02-27T15:54:27.096 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2021-02-27T15:54:27.097 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="8" 2021-02-27T15:54:27.097 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el8" 2021-02-27T15:54:27.097 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-02-27T15:54:27.097 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2021-02-27T15:54:27.098 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-02-27T15:54:27.098 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://www.centos.org/" 2021-02-27T15:54:27.098 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-27T15:54:27.098 INFO:teuthology.orchestra.run.smithi153.stdout: 2021-02-27T15:54:27.098 INFO:teuthology.orchestra.run.smithi153.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-02-27T15:54:27.099 INFO:teuthology.orchestra.run.smithi153.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-02-27T15:54:27.099 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-27T15:54:27.099 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-02-27T15:54:27.099 INFO:teuthology.orchestra.run.smithi153.stdout: 2021-02-27T15:54:27.100 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2021-02-27T15:54:27.182 INFO:teuthology.dispatcher.supervisor:Running job 5918478 2021-02-27T15:54:27.183 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-02-27_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/5918478 --name teuthology-2021-02-27_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi --description upgrade:octopus-x/stress-split/{0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export distro$/{centos_latest} mon_election/classic} -- /home/teuthworker/archive/teuthology-2021-02-27_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/5918478/orig.config.yaml 2021-02-27T15:54:27.187 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-02-27_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/5918478 2021-02-27T15:54:27.188 INFO:teuthology.dispatcher.supervisor:Job PID: 10205 2021-02-27T15:54:27.188 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-02-28T03:55:17.627 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-02-28T03:55:17.902 INFO:teuthology.kill:Killing Pids: {10205} 2021-02-28T03:55:19.056 INFO:teuthology.kill:Nuking machines: ['smithi075', 'smithi153'] 2021-02-28T03:55:19.974 INFO:teuthology.kill:2021-02-28 03:55:19,974.974 INFO:teuthology.nuke:targets: 2021-02-28T03:55:19.975 INFO:teuthology.kill: smithi075.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmgdemdmy4GkS1YO2wm7j8brpzbZCis6vABbIs9pIJ7EKIUaIYspmRYvNrSfFFKK4Sbj9Vh6CFf/pfzmwN1tt6jmcZr/ZGkxKMOKV+R+jNdt0BT1Chpa/46DII1EXf0GaleLd5ohKJFk7vx34himd++sNB6k9SbauO43fUz6yqb0s7nT5jiAyKXJo1CxC2yrwAFs/NjOsHE9U3i8kFbnV47Wp8H7nGjgE57TgkdGBJMjYfzM+Jq8JUtj/TdyPOiIfZ8Ywn0oRDuCzXmq0+3RuGSHxiIrZ4Y2DqE7OPaIQjqBHifjKMWdq+6HoKX4rz6j8dCamDS9ipdww9tP6sKDHZQgtyIojSwpSqHUQGJEsmFhtbnLC6od7U2DLRAzUJLUfxflUsbtAIABHzHp4steha5aCXtHky49g8CCYNNmgilKzp3k5kspBa5RhvDoJNNZL6Sp9EuRdpsgdnDjmb5eOgWzi2IoPlx7aSeBDq5Qd27kRZL/FL0mn0RMZjUA/+408= 2021-02-28T03:55:19.975 INFO:teuthology.kill: smithi153.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwCXyvxbAEdtVyTJ+mFfApQhNbFv4MYjwHJHcus4o7PXlp3c0xZfJslB4xHWoNnPcXPE3Qr3t0f859bzxzzUpF6iNqOqKnTYxMVHxTfpUb09A7hmVzW5GoShpo985lEJJ2V0x7DZ0zE7l3PokGlfeQQxcV4WSF1jNE36PdmrmwLH/5qUVb8uIKhzX0wsrwB4N5IdWLA118tvzpJdelyq9pzMAlE1MYedz2mWhmYlDFmJmrPU4HlCQxkjdhizL0fGLwWMc+IeVkXwLGzzrLN4Db/LzAhV3IqUlqUDjCvjdjWhL1/3mNO9vzJN/70aVjCnJ5UbamF+0xJwrZ0b6NCXdcTp6UkzeAVDzcuVuXxl8IPy8LBB+L9z3AOhZ9kQzsewkOzvJIK36Q6P2mVkGQNaW4nsLnwcWyEnYRImPyWC2OvbXvni7cqmjR10L+Wce2aS+tdkIwSbQaYWwoN63DWCRrsv7+qy/CUC+HM+3jTjrVRvPzy8739dM5blINVpaD6eE= 2021-02-28T03:55:19.975 INFO:teuthology.kill:2021-02-28 03:55:19,974.974 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-28T03:55:19.977 INFO:teuthology.kill:2021-02-28 03:55:19,977.977 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-28T03:55:20.046 INFO:teuthology.kill:2021-02-28 03:55:20,046.046 INFO:teuthology.orchestra.console:Power off smithi075 2021-02-28T03:55:20.061 INFO:teuthology.kill:2021-02-28 03:55:20,060.060 INFO:teuthology.orchestra.console:Power off smithi153 2021-02-28T03:55:28.416 INFO:teuthology.kill:2021-02-28 03:55:28,416.416 INFO:teuthology.orchestra.console:Power off for smithi075 completed 2021-02-28T03:55:32.539 INFO:teuthology.kill:2021-02-28 03:55:32,538.538 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2021-02-28T03:55:32.846 INFO:teuthology.task.internal:roles: ubuntu@smithi075.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2021-02-28T03:55:32.847 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2021-02-28T03:55:32.847 INFO:teuthology.misc:Compressing logs... 2021-02-28T03:55:32.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-28T03:55:51.366 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7a21eca3942b97058be0400a6c70d71b16304bdb/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-02-28T03:55:51.453 INFO:teuthology.kill:No teuthology processes running 2021-02-28T03:55:52.623 INFO:teuthology.kill:Nuking machines: ['smithi075', 'smithi153'] 2021-02-28T03:55:53.612 INFO:teuthology.kill:2021-02-28 03:55:53,612.612 INFO:teuthology.nuke:targets: 2021-02-28T03:55:53.613 INFO:teuthology.kill: smithi075.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmgdemdmy4GkS1YO2wm7j8brpzbZCis6vABbIs9pIJ7EKIUaIYspmRYvNrSfFFKK4Sbj9Vh6CFf/pfzmwN1tt6jmcZr/ZGkxKMOKV+R+jNdt0BT1Chpa/46DII1EXf0GaleLd5ohKJFk7vx34himd++sNB6k9SbauO43fUz6yqb0s7nT5jiAyKXJo1CxC2yrwAFs/NjOsHE9U3i8kFbnV47Wp8H7nGjgE57TgkdGBJMjYfzM+Jq8JUtj/TdyPOiIfZ8Ywn0oRDuCzXmq0+3RuGSHxiIrZ4Y2DqE7OPaIQjqBHifjKMWdq+6HoKX4rz6j8dCamDS9ipdww9tP6sKDHZQgtyIojSwpSqHUQGJEsmFhtbnLC6od7U2DLRAzUJLUfxflUsbtAIABHzHp4steha5aCXtHky49g8CCYNNmgilKzp3k5kspBa5RhvDoJNNZL6Sp9EuRdpsgdnDjmb5eOgWzi2IoPlx7aSeBDq5Qd27kRZL/FL0mn0RMZjUA/+408= 2021-02-28T03:55:53.614 INFO:teuthology.kill: smithi153.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwCXyvxbAEdtVyTJ+mFfApQhNbFv4MYjwHJHcus4o7PXlp3c0xZfJslB4xHWoNnPcXPE3Qr3t0f859bzxzzUpF6iNqOqKnTYxMVHxTfpUb09A7hmVzW5GoShpo985lEJJ2V0x7DZ0zE7l3PokGlfeQQxcV4WSF1jNE36PdmrmwLH/5qUVb8uIKhzX0wsrwB4N5IdWLA118tvzpJdelyq9pzMAlE1MYedz2mWhmYlDFmJmrPU4HlCQxkjdhizL0fGLwWMc+IeVkXwLGzzrLN4Db/LzAhV3IqUlqUDjCvjdjWhL1/3mNO9vzJN/70aVjCnJ5UbamF+0xJwrZ0b6NCXdcTp6UkzeAVDzcuVuXxl8IPy8LBB+L9z3AOhZ9kQzsewkOzvJIK36Q6P2mVkGQNaW4nsLnwcWyEnYRImPyWC2OvbXvni7cqmjR10L+Wce2aS+tdkIwSbQaYWwoN63DWCRrsv7+qy/CUC+HM+3jTjrVRvPzy8739dM5blINVpaD6eE= 2021-02-28T03:55:53.639 INFO:teuthology.kill:2021-02-28 03:55:53,637.637 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-28T03:55:53.642 INFO:teuthology.kill:2021-02-28 03:55:53,642.642 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-28T03:55:53.704 INFO:teuthology.kill:2021-02-28 03:55:53,703.703 INFO:teuthology.orchestra.console:Power off smithi075 2021-02-28T03:55:53.721 INFO:teuthology.kill:2021-02-28 03:55:53,721.721 INFO:teuthology.orchestra.console:Power off smithi153 2021-02-28T03:55:53.854 INFO:teuthology.kill:2021-02-28 03:55:53,854.854 INFO:teuthology.orchestra.console:Power off for smithi075 completed 2021-02-28T03:55:53.867 INFO:teuthology.kill:2021-02-28 03:55:53,867.867 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2021-02-28T03:55:54.004 INFO:teuthology.kill:2021-02-28 03:55:54,004.004 INFO:teuthology.lock.ops:unlocked smithi075.front.sepia.ceph.com 2021-02-28T03:55:54.011 INFO:teuthology.kill:2021-02-28 03:55:54,007.007 INFO:teuthology.lock.ops:unlocked smithi153.front.sepia.ceph.com 2021-02-28T03:57:54.362 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-02-28T03:57:54.481 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-02-28T03:57:54.481 INFO:teuthology.nuke:Checking targets against current locks 2021-02-28T03:57:54.566 INFO:teuthology.nuke:Not nuking smithi153.front.sepia.ceph.com because description doesn't match 2021-02-28T03:57:54.567 INFO:teuthology.nuke:Not nuking smithi075.front.sepia.ceph.com because description doesn't match