2021-03-06T18:11:09.960 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-06T18:11:10.056 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi002.front.sepia.ceph.com' reimaging 2021-03-06T18:11:10.058 INFO:teuthology.lock.ops:Updating [ubuntu@smithi002.front.sepia.ceph.com]: reset os type and version on server 2021-03-06T18:11:10.058 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2021-03-06T18:11:10.078 INFO:teuthology.lock.ops:Node 'ubuntu@smithi002.front.sepia.ceph.com' reimaging is complete 2021-03-06T18:11:10.079 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi012.front.sepia.ceph.com' reimaging 2021-03-06T18:11:10.079 INFO:teuthology.lock.ops:Updating [ubuntu@smithi012.front.sepia.ceph.com]: reset os type and version on server 2021-03-06T18:11:10.079 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2021-03-06T18:11:10.094 INFO:teuthology.lock.ops:Node 'ubuntu@smithi012.front.sepia.ceph.com' reimaging is complete 2021-03-06T18:11:10.283 INFO:teuthology.provision.fog.smithi002:Scheduling deploy of centos 8.2 2021-03-06T18:11:10.290 INFO:teuthology.provision.fog.smithi012:Scheduling deploy of centos 8.2 2021-03-06T18:11:10.603 INFO:teuthology.orchestra.console:Power off smithi002 2021-03-06T18:11:10.621 INFO:teuthology.orchestra.console:Power off smithi012 2021-03-06T18:11:23.093 INFO:teuthology.orchestra.console:Power off for smithi002 completed 2021-03-06T18:11:23.107 INFO:teuthology.orchestra.console:Power off for smithi012 completed 2021-03-06T18:11:23.195 INFO:teuthology.orchestra.console:Power on smithi002 2021-03-06T18:11:23.208 INFO:teuthology.orchestra.console:Power on smithi012 2021-03-06T18:11:31.564 INFO:teuthology.orchestra.console:Power on for smithi002 completed 2021-03-06T18:11:31.570 INFO:teuthology.orchestra.console:Power on for smithi012 completed 2021-03-06T18:11:31.666 INFO:teuthology.provision.fog.smithi002:Waiting for deploy to finish 2021-03-06T18:11:31.672 INFO:teuthology.provision.fog.smithi012:Waiting for deploy to finish 2021-03-06T18:15:39.288 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-06T18:15:39.624 INFO:teuthology.provision.fog.smithi002:Node is ready 2021-03-06T18:15:39.642 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002.front.sepia.ceph.com 2021-03-06T18:15:39.698 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2 smithi002.front.sepia.ceph.com smithi002 2021-03-06T18:15:40.041 INFO:teuthology.provision.fog.smithi002:Deploy complete! 2021-03-06T18:16:45.722 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-06T18:16:46.050 INFO:teuthology.provision.fog.smithi012:Node is ready 2021-03-06T18:16:46.065 INFO:teuthology.orchestra.run.smithi012.stdout:smithi012.front.sepia.ceph.com 2021-03-06T18:16:46.120 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12 smithi012.front.sepia.ceph.com smithi012 2021-03-06T18:16:46.472 INFO:teuthology.provision.fog.smithi012:Deploy complete! 2021-03-06T18:16:46.657 INFO:teuthology.lock.ops:Checking smithi002.front.sepia.ceph.com 2021-03-06T18:16:46.658 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-06T18:16:46.676 INFO:teuthology.lock.ops:Checking smithi012.front.sepia.ceph.com 2021-03-06T18:16:46.676 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-06T18:16:46.691 INFO:teuthology.lock.ops:Updating [smithi002.front.sepia.ceph.com]: set os type and version on server 2021-03-06T18:16:46.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-06T18:16:46.981 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2021-03-06T18:16:47.036 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="CentOS Linux" 2021-03-06T18:16:47.036 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="8 (Core)" 2021-03-06T18:16:47.037 INFO:teuthology.orchestra.run.smithi002.stdout:ID="centos" 2021-03-06T18:16:47.037 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="rhel fedora" 2021-03-06T18:16:47.037 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="8" 2021-03-06T18:16:47.037 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el8" 2021-03-06T18:16:47.038 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-03-06T18:16:47.038 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2021-03-06T18:16:47.038 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-03-06T18:16:47.038 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://www.centos.org/" 2021-03-06T18:16:47.039 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-06T18:16:47.039 INFO:teuthology.orchestra.run.smithi002.stdout: 2021-03-06T18:16:47.039 INFO:teuthology.orchestra.run.smithi002.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-03-06T18:16:47.039 INFO:teuthology.orchestra.run.smithi002.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-03-06T18:16:47.039 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-03-06T18:16:47.040 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-03-06T18:16:47.040 INFO:teuthology.orchestra.run.smithi002.stdout: 2021-03-06T18:16:47.041 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2021-03-06T18:16:47.059 INFO:teuthology.lock.ops:Updating [smithi012.front.sepia.ceph.com]: set os type and version on server 2021-03-06T18:16:47.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-06T18:16:47.340 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2021-03-06T18:16:47.395 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="CentOS Linux" 2021-03-06T18:16:47.395 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="8 (Core)" 2021-03-06T18:16:47.395 INFO:teuthology.orchestra.run.smithi012.stdout:ID="centos" 2021-03-06T18:16:47.395 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="rhel fedora" 2021-03-06T18:16:47.396 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="8" 2021-03-06T18:16:47.396 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el8" 2021-03-06T18:16:47.396 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-03-06T18:16:47.396 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2021-03-06T18:16:47.397 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-03-06T18:16:47.397 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://www.centos.org/" 2021-03-06T18:16:47.397 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-06T18:16:47.397 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-06T18:16:47.398 INFO:teuthology.orchestra.run.smithi012.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-03-06T18:16:47.399 INFO:teuthology.orchestra.run.smithi012.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-03-06T18:16:47.399 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-03-06T18:16:47.399 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-03-06T18:16:47.400 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-06T18:16:47.400 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2021-03-06T18:16:47.500 INFO:teuthology.dispatcher.supervisor:Running job 5942409 2021-03-06T18:16:47.501 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-03-06_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/5942409 --name teuthology-2021-03-06_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi --description upgrade:octopus-x/stress-split/{0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench distro$/{centos_latest} mon_election/connectivity} -- /home/teuthworker/archive/teuthology-2021-03-06_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/5942409/orig.config.yaml 2021-03-06T18:16:47.531 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-03-06_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/5942409 2021-03-06T18:16:47.532 INFO:teuthology.dispatcher.supervisor:Job PID: 27934 2021-03-06T18:16:47.532 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-07T06:17:38.407 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-07T06:17:38.698 INFO:teuthology.kill:Killing Pids: {27934} 2021-03-07T06:17:40.806 INFO:teuthology.kill:Nuking machines: ['smithi002', 'smithi012'] 2021-03-07T06:17:42.804 INFO:teuthology.kill:2021-03-07 06:17:42,803.803 INFO:teuthology.nuke:targets: 2021-03-07T06:17:42.806 INFO:teuthology.kill: smithi002.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+7a1GDBqLoC6ZVFJDpIDUeNvfXrhYVNAzOjdnEBTPYIfcVeiwmQHVi2qkuN9g4CVpyvQMRYR5IsDk1HcgEhvfthuQdv8IThQxudYZpnLx2GV8OCyggCzp1LDFu3Ms1g82/gBDivqM6WX71f9wqBtZKsBlf35RLXboUmIjTEYHnGkmKd+9n5RIrnWVk/4Z7y8MB66+TGoSjo+XUUKydbRxMbTajan2kNk9HXtGvhkGrfHFSMvgN+4BC6ZhdMZQjdHCm12F6OEGM8lJ7w+aJOcJ4ji8FYN1MNFtsvv3XwoF1SvAl6jEu5Bv77ETeUJwQ3TfEzdZFDvv+GMw1fxUN1SeTqGI5X0FDswCieRuc1Ah1AvOXRuNkGFc2Zny8HVN9n/wNI29VWqIsouVXp69pg4uPKsJF3yVKzgVb5epr0DP3nQVyrY+XlK8MwFG1XhWSpbH/yd+EU65dSdM/Ev7VF2ZQ+a3OwhDKnsgC1p2mzEl0lR8yTCEnHAIaXBKkRtZF6s= 2021-03-07T06:17:42.806 INFO:teuthology.kill: smithi012.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwV2554sCl4JaikrUFJsOUbFmbUU6MDJdHC8gyrHvVEsGfXj+IyCjtBxFny5CAu1gsWBVzzPgMu8ZVSns/ZcOkH3gJR+Okx12dMsCP6PcY0NupHQX9jUFFEZeNRmeHI8AmHUQ+FHrxaDcmoHbVIlk7R2C6WtRx7xZFc+6Kq8jgFqpAzG91wZn8/Iqxs2dnUZEwDpdL6oAHMu3vMs3wemSAmoH4aobVXwb2G47mJTkSI0PqvZJMU8kkP25SFXxrVSFUpeasWMYVDXSA7zNJWWiZK8EYcVwsNAJiVnwTuAZAKjJr8jh48cLvrj11rDI+uqInCZCWrguJKtccuDrSex2QJuZ71yxE2xEdxZGeuf+mSeQ9t6KlhcRVjJ+AnUKRSBCalMIrnxwhXZRBf3ysWXzZ3sHCvUHGpqZlIwQ2L0cusMd4k5n/qAa+9rUChHazvtrwXFyDCFABeAF5HRpPRpGHQ/z5br/YtCg+v75dRJfRIcaV30QznVj3bpTtOHqm2Hs= 2021-03-07T06:17:42.806 INFO:teuthology.kill:2021-03-07 06:17:42,805.805 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-07T06:17:42.810 INFO:teuthology.kill:2021-03-07 06:17:42,810.810 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-07T06:17:42.896 INFO:teuthology.kill:2021-03-07 06:17:42,896.896 INFO:teuthology.orchestra.console:Power off smithi012 2021-03-07T06:17:42.918 INFO:teuthology.kill:2021-03-07 06:17:42,917.917 INFO:teuthology.orchestra.console:Power off smithi002 2021-03-07T06:17:55.427 INFO:teuthology.kill:2021-03-07 06:17:55,426.426 INFO:teuthology.orchestra.console:Power off for smithi012 completed 2021-03-07T06:17:55.433 INFO:teuthology.kill:2021-03-07 06:17:55,433.433 INFO:teuthology.orchestra.console:Power off for smithi002 completed 2021-03-07T06:17:55.711 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2021-03-07T06:17:55.823 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2021-03-07T06:17:55.824 INFO:teuthology.misc:Compressing logs... 2021-03-07T06:17:55.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-07T06:18:55.889 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-07T06:18:56.064 INFO:teuthology.kill:No teuthology processes running 2021-03-07T06:18:58.629 INFO:teuthology.kill:Nuking machines: ['smithi002', 'smithi012'] 2021-03-07T06:19:00.703 INFO:teuthology.kill:2021-03-07 06:19:00,702.702 INFO:teuthology.nuke:targets: 2021-03-07T06:19:00.707 INFO:teuthology.kill: smithi002.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+7a1GDBqLoC6ZVFJDpIDUeNvfXrhYVNAzOjdnEBTPYIfcVeiwmQHVi2qkuN9g4CVpyvQMRYR5IsDk1HcgEhvfthuQdv8IThQxudYZpnLx2GV8OCyggCzp1LDFu3Ms1g82/gBDivqM6WX71f9wqBtZKsBlf35RLXboUmIjTEYHnGkmKd+9n5RIrnWVk/4Z7y8MB66+TGoSjo+XUUKydbRxMbTajan2kNk9HXtGvhkGrfHFSMvgN+4BC6ZhdMZQjdHCm12F6OEGM8lJ7w+aJOcJ4ji8FYN1MNFtsvv3XwoF1SvAl6jEu5Bv77ETeUJwQ3TfEzdZFDvv+GMw1fxUN1SeTqGI5X0FDswCieRuc1Ah1AvOXRuNkGFc2Zny8HVN9n/wNI29VWqIsouVXp69pg4uPKsJF3yVKzgVb5epr0DP3nQVyrY+XlK8MwFG1XhWSpbH/yd+EU65dSdM/Ev7VF2ZQ+a3OwhDKnsgC1p2mzEl0lR8yTCEnHAIaXBKkRtZF6s= 2021-03-07T06:19:00.707 INFO:teuthology.kill: smithi012.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwV2554sCl4JaikrUFJsOUbFmbUU6MDJdHC8gyrHvVEsGfXj+IyCjtBxFny5CAu1gsWBVzzPgMu8ZVSns/ZcOkH3gJR+Okx12dMsCP6PcY0NupHQX9jUFFEZeNRmeHI8AmHUQ+FHrxaDcmoHbVIlk7R2C6WtRx7xZFc+6Kq8jgFqpAzG91wZn8/Iqxs2dnUZEwDpdL6oAHMu3vMs3wemSAmoH4aobVXwb2G47mJTkSI0PqvZJMU8kkP25SFXxrVSFUpeasWMYVDXSA7zNJWWiZK8EYcVwsNAJiVnwTuAZAKjJr8jh48cLvrj11rDI+uqInCZCWrguJKtccuDrSex2QJuZ71yxE2xEdxZGeuf+mSeQ9t6KlhcRVjJ+AnUKRSBCalMIrnxwhXZRBf3ysWXzZ3sHCvUHGpqZlIwQ2L0cusMd4k5n/qAa+9rUChHazvtrwXFyDCFABeAF5HRpPRpGHQ/z5br/YtCg+v75dRJfRIcaV30QznVj3bpTtOHqm2Hs= 2021-03-07T06:19:00.738 INFO:teuthology.kill:2021-03-07 06:19:00,737.737 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-07T06:19:00.742 INFO:teuthology.kill:2021-03-07 06:19:00,741.741 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-07T06:19:00.831 INFO:teuthology.kill:2021-03-07 06:19:00,830.830 INFO:teuthology.orchestra.console:Power off smithi012 2021-03-07T06:19:00.886 INFO:teuthology.kill:2021-03-07 06:19:00,886.886 INFO:teuthology.orchestra.console:Power off smithi002 2021-03-07T06:19:01.108 INFO:teuthology.kill:2021-03-07 06:19:01,108.108 INFO:teuthology.orchestra.console:Power off for smithi012 completed 2021-03-07T06:19:01.126 INFO:teuthology.kill:2021-03-07 06:19:01,126.126 INFO:teuthology.orchestra.console:Power off for smithi002 completed 2021-03-07T06:19:01.262 INFO:teuthology.kill:2021-03-07 06:19:01,258.258 INFO:teuthology.lock.ops:unlocked smithi012.front.sepia.ceph.com 2021-03-07T06:19:01.274 INFO:teuthology.kill:2021-03-07 06:19:01,273.273 INFO:teuthology.lock.ops:unlocked smithi002.front.sepia.ceph.com 2021-03-07T06:21:01.722 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-03-07T06:21:01.911 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-07T06:21:01.912 INFO:teuthology.nuke:Checking targets against current locks 2021-03-07T06:21:01.998 INFO:teuthology.nuke:Not nuking smithi012.front.sepia.ceph.com because description doesn't match 2021-03-07T06:21:01.999 INFO:teuthology.nuke:Not nuking smithi002.front.sepia.ceph.com because description doesn't match