2021-04-21T15:46:14.034 INFO:root:teuthology version: 1.1.0-2713a3cd
2021-04-21T15:46:14.141 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi165.front.sepia.ceph.com' reimaging
2021-04-21T15:46:14.141 INFO:teuthology.lock.ops:Updating [ubuntu@smithi165.front.sepia.ceph.com]: reset os type and version on server
2021-04-21T15:46:14.142 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server
2021-04-21T15:46:14.160 INFO:teuthology.lock.ops:Node 'ubuntu@smithi165.front.sepia.ceph.com' reimaging is complete
2021-04-21T15:46:14.160 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi193.front.sepia.ceph.com' reimaging
2021-04-21T15:46:14.160 INFO:teuthology.lock.ops:Updating [ubuntu@smithi193.front.sepia.ceph.com]: reset os type and version on server
2021-04-21T15:46:14.160 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server
2021-04-21T15:46:14.176 INFO:teuthology.lock.ops:Node 'ubuntu@smithi193.front.sepia.ceph.com' reimaging is complete
2021-04-21T15:46:14.382 INFO:teuthology.provision.fog.smithi165:Scheduling deploy of centos 8.3
2021-04-21T15:46:14.403 INFO:teuthology.provision.fog.smithi193:Scheduling deploy of centos 8.3
2021-04-21T15:46:14.757 INFO:teuthology.orchestra.console:Power off smithi165
2021-04-21T15:46:14.776 INFO:teuthology.orchestra.console:Power off smithi193
2021-04-21T15:46:23.151 INFO:teuthology.orchestra.console:Power off for smithi193 completed
2021-04-21T15:46:23.252 INFO:teuthology.orchestra.console:Power on smithi193
2021-04-21T15:46:27.253 INFO:teuthology.orchestra.console:Power off for smithi165 completed
2021-04-21T15:46:27.355 INFO:teuthology.orchestra.console:Power on smithi165
2021-04-21T15:46:31.614 INFO:teuthology.orchestra.console:Power on for smithi165 completed
2021-04-21T15:46:31.644 INFO:teuthology.orchestra.console:Power on for smithi193 completed
2021-04-21T15:46:31.716 INFO:teuthology.provision.fog.smithi165:Waiting for deploy to finish
2021-04-21T15:46:31.745 INFO:teuthology.provision.fog.smithi193:Waiting for deploy to finish
2021-04-21T15:51:04.012 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-04-21T15:51:04.317 INFO:teuthology.provision.fog.smithi193:Node is ready
2021-04-21T15:51:04.373 INFO:teuthology.orchestra.run.smithi193.stdout:smithi193.front.sepia.ceph.com
2021-04-21T15:51:04.429 INFO:teuthology.orchestra.run.smithi193.stdout:172.21.15.193 smithi193.front.sepia.ceph.com smithi193
2021-04-21T15:51:04.763 INFO:teuthology.provision.fog.smithi193:Deploy complete!
2021-04-21T15:51:28.128 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-04-21T15:51:28.469 INFO:teuthology.provision.fog.smithi165:Node is ready
2021-04-21T15:51:28.484 INFO:teuthology.orchestra.run.smithi165.stdout:smithi165.front.sepia.ceph.com
2021-04-21T15:51:28.540 INFO:teuthology.orchestra.run.smithi165.stdout:172.21.15.165 smithi165.front.sepia.ceph.com smithi165
2021-04-21T15:51:28.863 INFO:teuthology.provision.fog.smithi165:Deploy complete!
2021-04-21T15:51:40.726 INFO:teuthology.lock.ops:Checking smithi165.front.sepia.ceph.com
2021-04-21T15:51:40.727 INFO:teuthology.lock.ops:New key found. Updating...
2021-04-21T15:51:40.753 INFO:teuthology.lock.ops:Checking smithi193.front.sepia.ceph.com
2021-04-21T15:51:40.754 INFO:teuthology.lock.ops:New key found. Updating...
2021-04-21T15:51:40.768 INFO:teuthology.lock.ops:Updating [smithi165.front.sepia.ceph.com]: set os type and version on server
2021-04-21T15:51:40.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-21T15:51:41.059 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64
2021-04-21T15:51:41.114 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Linux"
2021-04-21T15:51:41.115 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="8"
2021-04-21T15:51:41.115 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos"
2021-04-21T15:51:41.115 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora"
2021-04-21T15:51:41.115 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="8"
2021-04-21T15:51:41.115 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el8"
2021-04-21T15:51:41.115 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Linux 8"
2021-04-21T15:51:41.115 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31"
2021-04-21T15:51:41.116 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-04-21T15:51:41.116 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://centos.org/"
2021-04-21T15:51:41.116 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-04-21T15:51:41.116 INFO:teuthology.orchestra.run.smithi165.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-04-21T15:51:41.116 INFO:teuthology.orchestra.run.smithi165.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-04-21T15:51:41.117 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server
2021-04-21T15:51:41.149 INFO:teuthology.lock.ops:Updating [smithi193.front.sepia.ceph.com]: set os type and version on server
2021-04-21T15:51:41.149 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-21T15:51:41.440 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64
2021-04-21T15:51:41.494 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Linux"
2021-04-21T15:51:41.494 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="8"
2021-04-21T15:51:41.494 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos"
2021-04-21T15:51:41.494 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora"
2021-04-21T15:51:41.494 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="8"
2021-04-21T15:51:41.494 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el8"
2021-04-21T15:51:41.494 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Linux 8"
2021-04-21T15:51:41.494 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31"
2021-04-21T15:51:41.494 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-04-21T15:51:41.495 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/"
2021-04-21T15:51:41.495 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-04-21T15:51:41.495 INFO:teuthology.orchestra.run.smithi193.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-04-21T15:51:41.495 INFO:teuthology.orchestra.run.smithi193.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-04-21T15:51:41.495 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server
2021-04-21T15:51:41.572 INFO:teuthology.dispatcher.supervisor:Running job 6061348
2021-04-21T15:51:41.572 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/bin/teuthology -v --owner scheduled_ksirivad@teuthology --archive /home/teuthworker/archive/ksirivad-2021-04-21_14:41:00-rados-wip-pool-start-out-with-1-pg-distro-basic-smithi/6061348 --name ksirivad-2021-04-21_14:41:00-rados-wip-pool-start-out-with-1-pg-distro-basic-smithi --description rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.3-octopus 2-repo_digest/defaut 3-start-upgrade 4-wait mon_election/classic} -- /home/teuthworker/archive/ksirivad-2021-04-21_14:41:00-rados-wip-pool-start-out-with-1-pg-distro-basic-smithi/6061348/orig.config.yaml
2021-04-21T15:51:41.580 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/ksirivad-2021-04-21_14:41:00-rados-wip-pool-start-out-with-1-pg-distro-basic-smithi/6061348
2021-04-21T15:51:41.580 INFO:teuthology.dispatcher.supervisor:Job PID: 3939
2021-04-21T15:51:41.580 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-04-21T16:39:43.408 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/ksirivad-2021-04-21_14:41:00-rados-wip-pool-start-out-with-1-pg-distro-basic-smithi/jobs/ failed with status 504:
504 Gateway Time-out
504 Gateway Time-out
nginx/1.10.3 (Ubuntu)
2021-04-22T03:52:07.021 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-04-22T03:52:07.263 INFO:teuthology.kill:Killing Pids: {3939}
2021-04-22T03:52:08.704 INFO:teuthology.kill:Nuking machines: ['smithi165', 'smithi193']
2021-04-22T03:52:09.701 INFO:teuthology.kill:2021-04-22 03:52:09,701.701 INFO:teuthology.nuke:targets:
2021-04-22T03:52:09.701 INFO:teuthology.kill: smithi165.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDD/VTMRW2igeKzE6lqMKdU4iem3B2jtFZfrHn68R0rlWZQLfMeUm2I8Q2oMBWI2UpcnWfZTgJwzwVDoP97JQCNx1UB3mYuQ1XkE/znUh9t1z/Mx6NgKqY9XHf2uOCYyPQgVatpsisn7+Bo9rlTpbTZwCwPW2t+eiXa3z2Jq0wjzZCa3Vb+kJYl8ZSGrK5Kq3jdeSSKen7293Cm/AZvDzDV7GTtzIgMz9P+ewplqgVE/J0rAxhs4bT/0e0YGbj3a0LuZV9XxkpdXnxi4Wwvx7uVGDSTe08RxMzk1Of0dUioD8rlXTCrlwk/j55rpEoPTlA46NXdgw/5d0hMIufr0COvTe4p9aGsu00/6iSxkSeni/d3E+8wZxC/FI9QmPgZddtfXsIDGeaDOHubHbE4fYxJACwtvHFXQBJiwUJellF8kB2qM2AGL79tUanOT09udWEIAIIsGuA+yW36WhJh+j64XzX87jHXfJsP3N0ADE2fAB+lGQ28pFgxje1n9GhCT7M=
2021-04-22T03:52:09.701 INFO:teuthology.kill: smithi193.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQiUe+U4dhOHjz2+BnuhXnu730kTSlWLtgD7f1kt8SKB7BtMdzFqTpLuXKejWtTNhK3ADoe+tKggIN2UmbrUsCrfmC3SwIOvk66TmsKOoN4Ld0PmjfvGZiXpDEe9TBywWObNLnH2Itv3tCPY4p2PWysM+BLJeHgcmBde49jDwHX454q81DCYoqfOTt6CkveMATQbIXO1AahWmPdbg1b3GxzQeyKHdEh0JSObgTjRk4CZOZsS15JrdpGDVQcigEYQ2jXF6Bk/XHJcSUx/MngyMm0tjkX/jm5qvYgmPjpSJHpMBzFum1NCAvT1KvkiD03es6+ii1ldq0BrByQ7HTPCuZpmy03BBZ+Beeu9y/zSy4z7qjx414AiqFstbeALmYAK1A0lZETfHvXK5fievZWffQP4Z/afMColBpChltLZOz0rmTvkC2S02HkqNKT8IPXGktfnChUeq0UZVhih+1jqKviujlCjjEYhRUu2MDZz704YWOtGKWqP13Vy186KkaGQ0=
2021-04-22T03:52:09.701 INFO:teuthology.kill:2021-04-22 03:52:09,701.701 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-22T03:52:09.703 INFO:teuthology.kill:2021-04-22 03:52:09,703.703 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-22T03:52:09.776 INFO:teuthology.kill:2021-04-22 03:52:09,776.776 INFO:teuthology.orchestra.console:Power off smithi165
2021-04-22T03:52:09.789 INFO:teuthology.kill:2021-04-22 03:52:09,789.789 INFO:teuthology.orchestra.console:Power off smithi193
2021-04-22T03:52:18.150 INFO:teuthology.kill:2021-04-22 03:52:18,149.149 INFO:teuthology.orchestra.console:Power off for smithi193 completed
2021-04-22T03:52:22.256 INFO:teuthology.kill:2021-04-22 03:52:22,256.256 INFO:teuthology.orchestra.console:Power off for smithi165 completed
2021-04-22T03:52:22.477 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
2021-04-22T03:52:22.477 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a']
2021-04-22T03:52:22.477 INFO:teuthology.misc:Compressing logs...
2021-04-22T03:52:22.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-22T03:53:22.541 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
teuth_config.archive_base, job_config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
compress_logs(ctx, log_path)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/misc.py", line 1378, in compress_logs
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/remote.py", line 215, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 434, in run
transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-04-22T03:53:22.601 INFO:teuthology.kill:No teuthology processes running
2021-04-22T03:53:24.779 INFO:teuthology.kill:Nuking machines: ['smithi165', 'smithi193']
2021-04-22T03:53:25.880 INFO:teuthology.kill:2021-04-22 03:53:25,880.880 INFO:teuthology.nuke:targets:
2021-04-22T03:53:25.881 INFO:teuthology.kill: smithi165.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDD/VTMRW2igeKzE6lqMKdU4iem3B2jtFZfrHn68R0rlWZQLfMeUm2I8Q2oMBWI2UpcnWfZTgJwzwVDoP97JQCNx1UB3mYuQ1XkE/znUh9t1z/Mx6NgKqY9XHf2uOCYyPQgVatpsisn7+Bo9rlTpbTZwCwPW2t+eiXa3z2Jq0wjzZCa3Vb+kJYl8ZSGrK5Kq3jdeSSKen7293Cm/AZvDzDV7GTtzIgMz9P+ewplqgVE/J0rAxhs4bT/0e0YGbj3a0LuZV9XxkpdXnxi4Wwvx7uVGDSTe08RxMzk1Of0dUioD8rlXTCrlwk/j55rpEoPTlA46NXdgw/5d0hMIufr0COvTe4p9aGsu00/6iSxkSeni/d3E+8wZxC/FI9QmPgZddtfXsIDGeaDOHubHbE4fYxJACwtvHFXQBJiwUJellF8kB2qM2AGL79tUanOT09udWEIAIIsGuA+yW36WhJh+j64XzX87jHXfJsP3N0ADE2fAB+lGQ28pFgxje1n9GhCT7M=
2021-04-22T03:53:25.881 INFO:teuthology.kill: smithi193.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQiUe+U4dhOHjz2+BnuhXnu730kTSlWLtgD7f1kt8SKB7BtMdzFqTpLuXKejWtTNhK3ADoe+tKggIN2UmbrUsCrfmC3SwIOvk66TmsKOoN4Ld0PmjfvGZiXpDEe9TBywWObNLnH2Itv3tCPY4p2PWysM+BLJeHgcmBde49jDwHX454q81DCYoqfOTt6CkveMATQbIXO1AahWmPdbg1b3GxzQeyKHdEh0JSObgTjRk4CZOZsS15JrdpGDVQcigEYQ2jXF6Bk/XHJcSUx/MngyMm0tjkX/jm5qvYgmPjpSJHpMBzFum1NCAvT1KvkiD03es6+ii1ldq0BrByQ7HTPCuZpmy03BBZ+Beeu9y/zSy4z7qjx414AiqFstbeALmYAK1A0lZETfHvXK5fievZWffQP4Z/afMColBpChltLZOz0rmTvkC2S02HkqNKT8IPXGktfnChUeq0UZVhih+1jqKviujlCjjEYhRUu2MDZz704YWOtGKWqP13Vy186KkaGQ0=
2021-04-22T03:53:25.904 INFO:teuthology.kill:2021-04-22 03:53:25,904.904 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-22T03:53:25.917 INFO:teuthology.kill:2021-04-22 03:53:25,917.917 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-22T03:53:25.989 INFO:teuthology.kill:2021-04-22 03:53:25,988.988 INFO:teuthology.orchestra.console:Power off smithi165
2021-04-22T03:53:26.005 INFO:teuthology.kill:2021-04-22 03:53:26,005.005 INFO:teuthology.orchestra.console:Power off smithi193
2021-04-22T03:53:26.136 INFO:teuthology.kill:2021-04-22 03:53:26,136.136 INFO:teuthology.orchestra.console:Power off for smithi165 completed
2021-04-22T03:53:26.152 INFO:teuthology.kill:2021-04-22 03:53:26,152.152 INFO:teuthology.orchestra.console:Power off for smithi193 completed
2021-04-22T03:53:26.294 INFO:teuthology.kill:2021-04-22 03:53:26,293.293 INFO:teuthology.lock.ops:unlocked smithi165.front.sepia.ceph.com
2021-04-22T03:53:26.316 INFO:teuthology.kill:2021-04-22 03:53:26,316.316 INFO:teuthology.lock.ops:unlocked smithi193.front.sepia.ceph.com
2021-04-22T03:55:26.738 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-04-22T03:55:26.815 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-04-22T03:55:26.815 INFO:teuthology.nuke:Checking targets against current locks
2021-04-22T03:55:26.887 INFO:teuthology.nuke:Not nuking smithi165.front.sepia.ceph.com because description doesn't match
2021-04-22T03:55:26.888 INFO:teuthology.nuke:Not nuking smithi193.front.sepia.ceph.com because description doesn't match