2021-06-22T12:08:32.204 INFO:root:teuthology version: 1.1.0-26631ede
2021-06-22T12:08:32.290 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi096.front.sepia.ceph.com' reimaging
2021-06-22T12:08:32.291 INFO:teuthology.lock.ops:Updating [ubuntu@smithi096.front.sepia.ceph.com]: reset os type and version on server
2021-06-22T12:08:32.291 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server
2021-06-22T12:08:32.305 INFO:teuthology.lock.ops:Node 'ubuntu@smithi096.front.sepia.ceph.com' reimaging is complete
2021-06-22T12:08:32.305 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi163.front.sepia.ceph.com' reimaging
2021-06-22T12:08:32.306 INFO:teuthology.lock.ops:Updating [ubuntu@smithi163.front.sepia.ceph.com]: reset os type and version on server
2021-06-22T12:08:32.306 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server
2021-06-22T12:08:32.319 INFO:teuthology.lock.ops:Node 'ubuntu@smithi163.front.sepia.ceph.com' reimaging is complete
2021-06-22T12:08:32.540 INFO:teuthology.provision.fog.smithi096:Scheduling deploy of centos 8.2
2021-06-22T12:08:32.557 INFO:teuthology.provision.fog.smithi163:Scheduling deploy of centos 8.2
2021-06-22T12:08:32.860 INFO:teuthology.orchestra.console:Power off smithi096
2021-06-22T12:08:32.874 INFO:teuthology.orchestra.console:Power off smithi163
2021-06-22T12:08:41.230 INFO:teuthology.orchestra.console:Power off for smithi096 completed
2021-06-22T12:08:41.331 INFO:teuthology.orchestra.console:Power on smithi096
2021-06-22T12:08:45.346 INFO:teuthology.orchestra.console:Power off for smithi163 completed
2021-06-22T12:08:45.449 INFO:teuthology.orchestra.console:Power on smithi163
2021-06-22T12:08:49.693 INFO:teuthology.orchestra.console:Power on for smithi096 completed
2021-06-22T12:08:49.794 INFO:teuthology.provision.fog.smithi096:Waiting for deploy to finish
2021-06-22T12:08:53.813 INFO:teuthology.orchestra.console:Power on for smithi163 completed
2021-06-22T12:08:53.914 INFO:teuthology.provision.fog.smithi163:Waiting for deploy to finish
2021-06-22T12:12:53.821 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-06-22T12:13:14.192 INFO:teuthology.provision.fog.smithi096:Node is ready
2021-06-22T12:13:14.210 INFO:teuthology.orchestra.run.smithi096.stdout:smithi096.front.sepia.ceph.com
2021-06-22T12:13:14.266 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96 smithi096.front.sepia.ceph.com smithi096
2021-06-22T12:13:14.623 INFO:teuthology.provision.fog.smithi096:Deploy complete!
2021-06-22T12:14:20.796 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-06-22T12:14:21.111 INFO:teuthology.provision.fog.smithi163:Node is ready
2021-06-22T12:14:21.125 INFO:teuthology.orchestra.run.smithi163.stdout:smithi163.front.sepia.ceph.com
2021-06-22T12:14:21.179 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.15.163 smithi163.front.sepia.ceph.com smithi163
2021-06-22T12:14:21.496 INFO:teuthology.provision.fog.smithi163:Deploy complete!
2021-06-22T12:14:21.611 INFO:teuthology.lock.ops:Checking smithi096.front.sepia.ceph.com
2021-06-22T12:14:21.612 INFO:teuthology.lock.ops:New key found. Updating...
2021-06-22T12:14:21.627 INFO:teuthology.lock.ops:Checking smithi163.front.sepia.ceph.com
2021-06-22T12:14:21.628 INFO:teuthology.lock.ops:New key found. Updating...
2021-06-22T12:14:21.643 INFO:teuthology.lock.ops:Updating [smithi096.front.sepia.ceph.com]: set os type and version on server
2021-06-22T12:14:21.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-06-22T12:14:21.920 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64
2021-06-22T12:14:21.974 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Linux"
2021-06-22T12:14:21.974 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="8 (Core)"
2021-06-22T12:14:21.974 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos"
2021-06-22T12:14:21.974 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora"
2021-06-22T12:14:21.975 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="8"
2021-06-22T12:14:21.975 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el8"
2021-06-22T12:14:21.975 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Linux 8 (Core)"
2021-06-22T12:14:21.975 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31"
2021-06-22T12:14:21.975 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-06-22T12:14:21.976 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://www.centos.org/"
2021-06-22T12:14:21.976 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-06-22T12:14:21.976 INFO:teuthology.orchestra.run.smithi096.stdout:
2021-06-22T12:14:21.976 INFO:teuthology.orchestra.run.smithi096.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-06-22T12:14:21.976 INFO:teuthology.orchestra.run.smithi096.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-06-22T12:14:21.977 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="centos"
2021-06-22T12:14:21.977 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8"
2021-06-22T12:14:21.977 INFO:teuthology.orchestra.run.smithi096.stdout:
2021-06-22T12:14:21.978 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server
2021-06-22T12:14:21.992 INFO:teuthology.lock.ops:Updating [smithi163.front.sepia.ceph.com]: set os type and version on server
2021-06-22T12:14:21.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-06-22T12:14:22.258 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64
2021-06-22T12:14:22.311 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Linux"
2021-06-22T12:14:22.312 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="8 (Core)"
2021-06-22T12:14:22.312 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos"
2021-06-22T12:14:22.312 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora"
2021-06-22T12:14:22.312 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="8"
2021-06-22T12:14:22.313 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el8"
2021-06-22T12:14:22.313 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Linux 8 (Core)"
2021-06-22T12:14:22.313 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31"
2021-06-22T12:14:22.313 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-06-22T12:14:22.313 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://www.centos.org/"
2021-06-22T12:14:22.314 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-06-22T12:14:22.314 INFO:teuthology.orchestra.run.smithi163.stdout:
2021-06-22T12:14:22.314 INFO:teuthology.orchestra.run.smithi163.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-06-22T12:14:22.314 INFO:teuthology.orchestra.run.smithi163.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-06-22T12:14:22.315 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="centos"
2021-06-22T12:14:22.315 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8"
2021-06-22T12:14:22.315 INFO:teuthology.orchestra.run.smithi163.stdout:
2021-06-22T12:14:22.316 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server
2021-06-22T12:14:22.381 INFO:teuthology.dispatcher.supervisor:Running job 6184251
2021-06-22T12:14:22.382 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/virtualenv/bin/teuthology -v --owner scheduled_ideepika@teuthology --archive /home/teuthworker/archive/ideepika-2021-06-22_12:06:52-rados-wip-yuri7-testing-2021-06-08-0747-octopus-distro-basic-smithi/6184251 --name ideepika-2021-06-22_12:06:52-rados-wip-yuri7-testing-2021-06-08-0747-octopus-distro-basic-smithi --description rados/cephadm/upgrade/{1-start-distro/1-start-centos_8 2-repo_digest/repo_digest 3-start-upgrade 4-wait fixed-2} -- /home/teuthworker/archive/ideepika-2021-06-22_12:06:52-rados-wip-yuri7-testing-2021-06-08-0747-octopus-distro-basic-smithi/6184251/orig.config.yaml
2021-06-22T12:14:22.385 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/ideepika-2021-06-22_12:06:52-rados-wip-yuri7-testing-2021-06-08-0747-octopus-distro-basic-smithi/6184251
2021-06-22T12:14:22.386 INFO:teuthology.dispatcher.supervisor:Job PID: 430
2021-06-22T12:14:22.386 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-06-22T15:02:32.339 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/ideepika-2021-06-22_12:06:52-rados-wip-yuri7-testing-2021-06-08-0747-octopus-distro-basic-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-06-22T15:50:36.418 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/ideepika-2021-06-22_12:06:52-rados-wip-yuri7-testing-2021-06-08-0747-octopus-distro-basic-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-06-23T00:15:11.692 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-06-23T00:15:12.338 INFO:teuthology.kill:Killing Pids: {430}
2021-06-23T00:15:13.607 INFO:teuthology.kill:Nuking machines: ['smithi096', 'smithi163']
2021-06-23T00:15:14.430 INFO:teuthology.kill:2021-06-23 00:15:14,429.429 INFO:teuthology.nuke:targets:
2021-06-23T00:15:14.431 INFO:teuthology.kill: smithi096.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqER06cMei6ttQSqpQHDYEFKYzlrV9bMcsAlwRxlb+Ob1xNjj+T2MGtOdXx8n0Mx8zN4MnOqO6oWHNUBSAp2MaLSbFu7gtrV8Q9b43MgJxOM57yl667nXPg/yK2wVTeCJq4cF3ihK2fzKqa7GGtkbVm18WPJ8nzvJljvgrxvbr85ewHcX9MXHm/MJUlOpqmU0q1FA3kuY5XJUqgEukKSO3VkA9zNaeMNRSQZERZfrIbnmppTgMlw4bLNOFGMGsOFqkhwHzf2L3fIdmg5vTCqxI8x9PGxuMWpTkwRDKnqXhwIOyRJqRtXjoHuyCMds5Y5rZc+adgnhTolZNXu7xaSFnvY96OL1XzrHgG6eTJ1pHzwsuMadnok2+ogtIwJMMDgvtx46dWda6+9Rf4sS1l9+1jX7ZQfKss3f9roGHcefLx7dVb0PQxfxvOL4A1Nqxl/RT3U/+Ri5xm/qJ32jubejWkL9fvFBztOECKiD+U6LCLHPD64OAhnb+OCBGPplSAGk=
2021-06-23T00:15:14.432 INFO:teuthology.kill: smithi163.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2sNSVgXvL9DyGpKF+T3JtE61ohOG7ZlaLf89eN8yDCeTJ+ShSbdPSIcBr0X3WaopDWAYcFlLmGigCTl3ixeepytOCxEy/TMKD902fvIxx3n6Zsso/mJ2qGlO2xPE6xqvJOk4a7ZPhfmK/6vSNCcMxClf3BP9kBRilaYdrdZ9EBf7RPb/Y5ezac/fYW/bSwFAC5KGBR90hAV3UDm4jBK+WnydvU0e+qNfjNGJZuBy5tbpdogtutuxLJ+V74wynOiwEFoyPhvp8b+QBK3OJMvpDJlagH21i610VvNmTqADAab7AvHk6oDxCsXFoVbf8Q1gAG5Tdmu2uW5/2zDl4gNcLQxitg2Hdh7/6pPG1EWkomKuF4pFo5ak9wIraTfdFvOqshLDTKV9Gy0kZ1u5KCdvIR4tqFzPCYXq1FVhJVrhbgnNZw+DzdQYBqRWrkQm5e1lnPflrsWLCfzlZ+wVmNki38hjQgoTpds8unRHSz///3WckW4aMTfMMRmkLX5v0a0c=
2021-06-23T00:15:14.433 INFO:teuthology.kill:2021-06-23 00:15:14,429.429 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-06-23T00:15:14.433 INFO:teuthology.kill:2021-06-23 00:15:14,431.431 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-06-23T00:15:14.488 INFO:teuthology.kill:2021-06-23 00:15:14,487.487 INFO:teuthology.orchestra.console:Power off smithi163
2021-06-23T00:15:14.501 INFO:teuthology.kill:2021-06-23 00:15:14,501.501 INFO:teuthology.orchestra.console:Power off smithi096
2021-06-23T00:15:22.861 INFO:teuthology.kill:2021-06-23 00:15:22,861.861 INFO:teuthology.orchestra.console:Power off for smithi096 completed
2021-06-23T00:15:26.967 INFO:teuthology.kill:2021-06-23 00:15:26,967.967 INFO:teuthology.orchestra.console:Power off for smithi163 completed
2021-06-23T00:15:27.234 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
2021-06-23T00:15:27.237 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']
2021-06-23T00:15:27.244 INFO:teuthology.misc:Compressing logs...
2021-06-23T00:15:27.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-06-23T00:16:02.009 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
teuth_config.archive_base, job_config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
compress_logs(ctx, log_path)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/misc.py", line 1378, in compress_logs
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/cluster.py", line 64, in
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/remote.py", line 509, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/run.py", line 434, in run
transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-06-23T00:16:02.081 INFO:teuthology.kill:No teuthology processes running
2021-06-23T00:16:03.098 INFO:teuthology.kill:Nuking machines: ['smithi096', 'smithi163']
2021-06-23T00:16:03.957 INFO:teuthology.kill:2021-06-23 00:16:03,957.957 INFO:teuthology.nuke:targets:
2021-06-23T00:16:03.958 INFO:teuthology.kill: smithi096.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqER06cMei6ttQSqpQHDYEFKYzlrV9bMcsAlwRxlb+Ob1xNjj+T2MGtOdXx8n0Mx8zN4MnOqO6oWHNUBSAp2MaLSbFu7gtrV8Q9b43MgJxOM57yl667nXPg/yK2wVTeCJq4cF3ihK2fzKqa7GGtkbVm18WPJ8nzvJljvgrxvbr85ewHcX9MXHm/MJUlOpqmU0q1FA3kuY5XJUqgEukKSO3VkA9zNaeMNRSQZERZfrIbnmppTgMlw4bLNOFGMGsOFqkhwHzf2L3fIdmg5vTCqxI8x9PGxuMWpTkwRDKnqXhwIOyRJqRtXjoHuyCMds5Y5rZc+adgnhTolZNXu7xaSFnvY96OL1XzrHgG6eTJ1pHzwsuMadnok2+ogtIwJMMDgvtx46dWda6+9Rf4sS1l9+1jX7ZQfKss3f9roGHcefLx7dVb0PQxfxvOL4A1Nqxl/RT3U/+Ri5xm/qJ32jubejWkL9fvFBztOECKiD+U6LCLHPD64OAhnb+OCBGPplSAGk=
2021-06-23T00:16:03.958 INFO:teuthology.kill: smithi163.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2sNSVgXvL9DyGpKF+T3JtE61ohOG7ZlaLf89eN8yDCeTJ+ShSbdPSIcBr0X3WaopDWAYcFlLmGigCTl3ixeepytOCxEy/TMKD902fvIxx3n6Zsso/mJ2qGlO2xPE6xqvJOk4a7ZPhfmK/6vSNCcMxClf3BP9kBRilaYdrdZ9EBf7RPb/Y5ezac/fYW/bSwFAC5KGBR90hAV3UDm4jBK+WnydvU0e+qNfjNGJZuBy5tbpdogtutuxLJ+V74wynOiwEFoyPhvp8b+QBK3OJMvpDJlagH21i610VvNmTqADAab7AvHk6oDxCsXFoVbf8Q1gAG5Tdmu2uW5/2zDl4gNcLQxitg2Hdh7/6pPG1EWkomKuF4pFo5ak9wIraTfdFvOqshLDTKV9Gy0kZ1u5KCdvIR4tqFzPCYXq1FVhJVrhbgnNZw+DzdQYBqRWrkQm5e1lnPflrsWLCfzlZ+wVmNki38hjQgoTpds8unRHSz///3WckW4aMTfMMRmkLX5v0a0c=
2021-06-23T00:16:03.976 INFO:teuthology.kill:2021-06-23 00:16:03,976.976 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-06-23T00:16:03.979 INFO:teuthology.kill:2021-06-23 00:16:03,979.979 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-06-23T00:16:04.031 INFO:teuthology.kill:2021-06-23 00:16:04,031.031 INFO:teuthology.orchestra.console:Power off smithi163
2021-06-23T00:16:04.044 INFO:teuthology.kill:2021-06-23 00:16:04,043.043 INFO:teuthology.orchestra.console:Power off smithi096
2021-06-23T00:16:04.170 INFO:teuthology.kill:2021-06-23 00:16:04,170.170 INFO:teuthology.orchestra.console:Power off for smithi163 completed
2021-06-23T00:16:04.181 INFO:teuthology.kill:2021-06-23 00:16:04,181.181 INFO:teuthology.orchestra.console:Power off for smithi096 completed
2021-06-23T00:16:04.312 INFO:teuthology.kill:2021-06-23 00:16:04,311.311 INFO:teuthology.lock.ops:unlocked smithi163.front.sepia.ceph.com
2021-06-23T00:16:04.324 INFO:teuthology.kill:2021-06-23 00:16:04,323.323 INFO:teuthology.lock.ops:unlocked smithi096.front.sepia.ceph.com
2021-06-23T00:18:04.595 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15