2021-07-22T18:15:00.312 INFO:root:teuthology version: 1.1.0-b6f3efab
2021-07-22T18:15:00.493 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi063.front.sepia.ceph.com' reimaging
2021-07-22T18:15:00.494 INFO:teuthology.lock.ops:Updating [ubuntu@smithi063.front.sepia.ceph.com]: reset os type and version on server
2021-07-22T18:15:00.495 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server
2021-07-22T18:15:00.532 INFO:teuthology.lock.ops:Node 'ubuntu@smithi063.front.sepia.ceph.com' reimaging is complete
2021-07-22T18:15:00.532 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi093.front.sepia.ceph.com' reimaging
2021-07-22T18:15:00.533 INFO:teuthology.lock.ops:Updating [ubuntu@smithi093.front.sepia.ceph.com]: reset os type and version on server
2021-07-22T18:15:00.533 INFO:teuthology.lock.ops:Updating smithi093.front.sepia.ceph.com on lock server
2021-07-22T18:15:00.560 INFO:teuthology.lock.ops:Node 'ubuntu@smithi093.front.sepia.ceph.com' reimaging is complete
2021-07-22T18:15:00.769 INFO:teuthology.provision.fog.smithi063:Scheduling deploy of ubuntu 20.04
2021-07-22T18:15:00.777 INFO:teuthology.provision.fog.smithi093:Scheduling deploy of ubuntu 20.04
2021-07-22T18:15:01.146 INFO:teuthology.orchestra.console:Power off smithi093
2021-07-22T18:15:01.190 INFO:teuthology.orchestra.console:Power off smithi063
2021-07-22T18:15:09.598 INFO:teuthology.orchestra.console:Power off for smithi063 completed
2021-07-22T18:15:09.701 INFO:teuthology.orchestra.console:Power on smithi063
2021-07-22T18:15:13.650 INFO:teuthology.orchestra.console:Power off for smithi093 completed
2021-07-22T18:15:13.754 INFO:teuthology.orchestra.console:Power on smithi093
2021-07-22T18:15:18.022 INFO:teuthology.orchestra.console:Power on for smithi093 completed
2021-07-22T18:15:18.094 INFO:teuthology.orchestra.console:Power on for smithi063 completed
2021-07-22T18:15:18.125 INFO:teuthology.provision.fog.smithi093:Waiting for deploy to finish
2021-07-22T18:15:18.196 INFO:teuthology.provision.fog.smithi063:Waiting for deploy to finish
2021-07-22T18:20:18.665 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-07-22T18:20:19.091 INFO:teuthology.provision.fog.smithi063:Node is ready
2021-07-22T18:20:19.139 INFO:teuthology.orchestra.run.smithi063.stdout:smithi063.front.sepia.ceph.com
2021-07-22T18:20:19.147 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63 smithi063.front.sepia.ceph.com smithi063
2021-07-22T18:20:19.322 INFO:teuthology.provision.fog.smithi063:Deploy complete!
2021-07-22T18:20:21.630 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-07-22T18:20:22.040 INFO:teuthology.provision.fog.smithi093:Node is ready
2021-07-22T18:20:22.046 INFO:teuthology.orchestra.run.smithi093.stdout:smithi093.front.sepia.ceph.com
2021-07-22T18:20:22.053 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.15.93 smithi093.front.sepia.ceph.com smithi093
2021-07-22T18:20:22.278 INFO:teuthology.provision.fog.smithi093:Deploy complete!
2021-07-22T18:20:22.458 INFO:teuthology.lock.ops:Checking smithi063.front.sepia.ceph.com
2021-07-22T18:20:22.459 INFO:teuthology.lock.ops:New key found. Updating...
2021-07-22T18:20:22.475 INFO:teuthology.lock.ops:Checking smithi093.front.sepia.ceph.com
2021-07-22T18:20:22.476 INFO:teuthology.lock.ops:New key found. Updating...
2021-07-22T18:20:22.495 INFO:teuthology.lock.ops:Updating [smithi063.front.sepia.ceph.com]: set os type and version on server
2021-07-22T18:20:22.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-07-22T18:20:22.723 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64
2021-07-22T18:20:22.730 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="Ubuntu"
2021-07-22T18:20:22.730 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="20.04.2 LTS (Focal Fossa)"
2021-07-22T18:20:22.730 INFO:teuthology.orchestra.run.smithi063.stdout:ID=ubuntu
2021-07-22T18:20:22.731 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE=debian
2021-07-22T18:20:22.731 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS"
2021-07-22T18:20:22.731 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="20.04"
2021-07-22T18:20:22.731 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://www.ubuntu.com/"
2021-07-22T18:20:22.731 INFO:teuthology.orchestra.run.smithi063.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-07-22T18:20:22.732 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-07-22T18:20:22.732 INFO:teuthology.orchestra.run.smithi063.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-07-22T18:20:22.732 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_CODENAME=focal
2021-07-22T18:20:22.732 INFO:teuthology.orchestra.run.smithi063.stdout:UBUNTU_CODENAME=focal
2021-07-22T18:20:22.733 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server
2021-07-22T18:20:22.749 INFO:teuthology.lock.ops:Updating [smithi093.front.sepia.ceph.com]: set os type and version on server
2021-07-22T18:20:22.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-07-22T18:20:22.927 INFO:teuthology.orchestra.run.smithi093.stdout:x86_64
2021-07-22T18:20:22.934 INFO:teuthology.orchestra.run.smithi093.stdout:NAME="Ubuntu"
2021-07-22T18:20:22.934 INFO:teuthology.orchestra.run.smithi093.stdout:VERSION="20.04.2 LTS (Focal Fossa)"
2021-07-22T18:20:22.934 INFO:teuthology.orchestra.run.smithi093.stdout:ID=ubuntu
2021-07-22T18:20:22.934 INFO:teuthology.orchestra.run.smithi093.stdout:ID_LIKE=debian
2021-07-22T18:20:22.935 INFO:teuthology.orchestra.run.smithi093.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS"
2021-07-22T18:20:22.935 INFO:teuthology.orchestra.run.smithi093.stdout:VERSION_ID="20.04"
2021-07-22T18:20:22.935 INFO:teuthology.orchestra.run.smithi093.stdout:HOME_URL="https://www.ubuntu.com/"
2021-07-22T18:20:22.935 INFO:teuthology.orchestra.run.smithi093.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-07-22T18:20:22.936 INFO:teuthology.orchestra.run.smithi093.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-07-22T18:20:22.936 INFO:teuthology.orchestra.run.smithi093.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-07-22T18:20:22.936 INFO:teuthology.orchestra.run.smithi093.stdout:VERSION_CODENAME=focal
2021-07-22T18:20:22.936 INFO:teuthology.orchestra.run.smithi093.stdout:UBUNTU_CODENAME=focal
2021-07-22T18:20:22.937 INFO:teuthology.lock.ops:Updating smithi093.front.sepia.ceph.com on lock server
2021-07-22T18:20:23.027 INFO:teuthology.dispatcher.supervisor:Running job 6287303
2021-07-22T18:20:23.028 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/bin/teuthology -v --owner scheduled_nojha@teuthology --archive /home/teuthworker/archive/nojha-2021-07-22_18:00:50-rados-wip-sseshasa-testing-2021-07-14-1320-distro-basic-smithi/6287303 --name nojha-2021-07-22_18:00:50-rados-wip-sseshasa-testing-2021-07-14-1320-distro-basic-smithi --description rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/snaps-few-objects} -- /home/teuthworker/archive/nojha-2021-07-22_18:00:50-rados-wip-sseshasa-testing-2021-07-14-1320-distro-basic-smithi/6287303/orig.config.yaml
2021-07-22T18:20:23.032 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nojha-2021-07-22_18:00:50-rados-wip-sseshasa-testing-2021-07-14-1320-distro-basic-smithi/6287303
2021-07-22T18:20:23.033 INFO:teuthology.dispatcher.supervisor:Job PID: 22991
2021-07-22T18:20:23.033 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-07-23T04:51:07.113 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/nojha-2021-07-22_18:00:50-rados-wip-sseshasa-testing-2021-07-14-1320-distro-basic-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-07-23T06:21:14.324 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-07-23T06:21:14.560 INFO:teuthology.kill:Killing Pids: {22991}
2021-07-23T06:21:15.619 INFO:teuthology.kill:Nuking machines: ['smithi063', 'smithi093']
2021-07-23T06:21:16.535 INFO:teuthology.kill:2021-07-23 06:21:16,534.534 INFO:teuthology.nuke:targets:
2021-07-23T06:21:16.536 INFO:teuthology.kill: smithi063.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMw7wddsE8szqDntGuw/WMhGVbZgOivikWfQF7AtYhCu/QmJDXejc5CXFKoaZnVlumBYbH34xAEvIowatyMDNff9yCKwmohMALCs9oiSkkeZENAkXG+fxx0Jsjpqy+r0jfN+D0V92/gqcbjOkQn2DWb43qpU8DIk6+Ddu1Nj20QQNxrNg+FOWDpt5ZOpmrNvBJwO+Ic7VAwRE1Xyf7Wgg4ZlBEFyL73JyXFDTjSO93RQSF1FVfgtFZxarOrZwZ+MrD+e7f0hPEuWh+OdTrTqGd4YPiPaxIVHfX3F6dutTzWhoA1K/Uhp4JvzfofHBVBAXnaWxD4zFsmdcpcq1hWqAvxTvAtt+r8TNQk16VYKhjaBr59u4tYnb9Uk9H9DEuSAoVaUoT6NXqX9exSDI5TCACbX2naynAZ9sZq2NFLw+nzF53enOzDyJdcMGbyq0Cfz3R6Gs+TVdWIiW+uCfKlhH4iCXIG1I5Gwn6DXEDYEJt9P+SmFgeOO4Kul6H4s/9r4k=
2021-07-23T06:21:16.536 INFO:teuthology.kill: smithi093.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3FFZwvwqptcf7+Rn59PotSQKFXctH17u7GUG9gzReIiS78fL9VwSFwwv+2n/s1CqQH0tuLoY04S9EIFlcOM6Eb2xLlo6WsyNmWfElN62nhg7kopBkVzBjYq4uFmYwFwtTWNN3tGaOv7bWGWV0jnyehbu5ZHUXzzDOuj4eHCm62MWUtKYD2+EjZupM4xxj1urqxlUpwhfPPG0f8WC+scwOut0xQavL9T9BqBdUfJ6hNDlvYynQXb0dhULvXj5SVcbQc9Cc3Ect3Ev3nfcgXBYyt1H9Bm2GKZ264THSuK3rP0x6VwnhaidQPUpQWj5FrfssMSrn05bv20kWGL9kHkfCh9vRf59d5S/vrhfLl2Pq0tv6jzDIQ/Zxf3OG3qByoyWXoSCongl/IgUHLh57E89ohmedCvGxsp0LtaKyGlANxI1fCR02sfm3sBKU76PBeYyyWhFOrgF2g/0NkFPxOCdsno+hGruMp9HPRtIg1IQdAnGBUWpjqxT4WnVkJVpkUb8=
2021-07-23T06:21:16.536 INFO:teuthology.kill:2021-07-23 06:21:16,535.535 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-07-23T06:21:16.538 INFO:teuthology.kill:2021-07-23 06:21:16,537.537 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-07-23T06:21:16.593 INFO:teuthology.kill:2021-07-23 06:21:16,593.593 INFO:teuthology.orchestra.console:Power off smithi063
2021-07-23T06:21:16.605 INFO:teuthology.kill:2021-07-23 06:21:16,605.605 INFO:teuthology.orchestra.console:Power off smithi093
2021-07-23T06:21:29.074 INFO:teuthology.kill:2021-07-23 06:21:29,073.073 INFO:teuthology.orchestra.console:Power off for smithi063 completed
2021-07-23T06:21:29.080 INFO:teuthology.kill:2021-07-23 06:21:29,080.080 INFO:teuthology.orchestra.console:Power off for smithi093 completed
2021-07-23T06:21:29.316 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2']
2021-07-23T06:21:29.316 INFO:teuthology.task.internal:roles: ubuntu@smithi093.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']
2021-07-23T06:21:29.317 INFO:teuthology.misc:Compressing logs...
2021-07-23T06:21:29.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-07-23T06:22:29.379 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
teuth_config.archive_base, job_config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
compress_logs(ctx, log_path)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/misc.py", line 1378, in compress_logs
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/orchestra/cluster.py", line 64, in
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/orchestra/remote.py", line 509, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/orchestra/run.py", line 434, in run
transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-07-23T06:22:29.432 INFO:teuthology.kill:No teuthology processes running
2021-07-23T06:22:30.502 INFO:teuthology.kill:Nuking machines: ['smithi063', 'smithi093']
2021-07-23T06:22:31.432 INFO:teuthology.kill:2021-07-23 06:22:31,432.432 INFO:teuthology.nuke:targets:
2021-07-23T06:22:31.433 INFO:teuthology.kill: smithi063.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMw7wddsE8szqDntGuw/WMhGVbZgOivikWfQF7AtYhCu/QmJDXejc5CXFKoaZnVlumBYbH34xAEvIowatyMDNff9yCKwmohMALCs9oiSkkeZENAkXG+fxx0Jsjpqy+r0jfN+D0V92/gqcbjOkQn2DWb43qpU8DIk6+Ddu1Nj20QQNxrNg+FOWDpt5ZOpmrNvBJwO+Ic7VAwRE1Xyf7Wgg4ZlBEFyL73JyXFDTjSO93RQSF1FVfgtFZxarOrZwZ+MrD+e7f0hPEuWh+OdTrTqGd4YPiPaxIVHfX3F6dutTzWhoA1K/Uhp4JvzfofHBVBAXnaWxD4zFsmdcpcq1hWqAvxTvAtt+r8TNQk16VYKhjaBr59u4tYnb9Uk9H9DEuSAoVaUoT6NXqX9exSDI5TCACbX2naynAZ9sZq2NFLw+nzF53enOzDyJdcMGbyq0Cfz3R6Gs+TVdWIiW+uCfKlhH4iCXIG1I5Gwn6DXEDYEJt9P+SmFgeOO4Kul6H4s/9r4k=
2021-07-23T06:22:31.434 INFO:teuthology.kill: smithi093.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3FFZwvwqptcf7+Rn59PotSQKFXctH17u7GUG9gzReIiS78fL9VwSFwwv+2n/s1CqQH0tuLoY04S9EIFlcOM6Eb2xLlo6WsyNmWfElN62nhg7kopBkVzBjYq4uFmYwFwtTWNN3tGaOv7bWGWV0jnyehbu5ZHUXzzDOuj4eHCm62MWUtKYD2+EjZupM4xxj1urqxlUpwhfPPG0f8WC+scwOut0xQavL9T9BqBdUfJ6hNDlvYynQXb0dhULvXj5SVcbQc9Cc3Ect3Ev3nfcgXBYyt1H9Bm2GKZ264THSuK3rP0x6VwnhaidQPUpQWj5FrfssMSrn05bv20kWGL9kHkfCh9vRf59d5S/vrhfLl2Pq0tv6jzDIQ/Zxf3OG3qByoyWXoSCongl/IgUHLh57E89ohmedCvGxsp0LtaKyGlANxI1fCR02sfm3sBKU76PBeYyyWhFOrgF2g/0NkFPxOCdsno+hGruMp9HPRtIg1IQdAnGBUWpjqxT4WnVkJVpkUb8=
2021-07-23T06:22:31.453 INFO:teuthology.kill:2021-07-23 06:22:31,453.453 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-07-23T06:22:31.456 INFO:teuthology.kill:2021-07-23 06:22:31,456.456 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-07-23T06:22:31.510 INFO:teuthology.kill:2021-07-23 06:22:31,510.510 INFO:teuthology.orchestra.console:Power off smithi093
2021-07-23T06:22:31.522 INFO:teuthology.kill:2021-07-23 06:22:31,522.522 INFO:teuthology.orchestra.console:Power off smithi063
2021-07-23T06:22:31.652 INFO:teuthology.kill:2021-07-23 06:22:31,652.652 INFO:teuthology.orchestra.console:Power off for smithi093 completed
2021-07-23T06:22:31.666 INFO:teuthology.kill:2021-07-23 06:22:31,665.665 INFO:teuthology.orchestra.console:Power off for smithi063 completed
2021-07-23T06:22:31.793 INFO:teuthology.kill:2021-07-23 06:22:31,792.792 INFO:teuthology.lock.ops:unlocked smithi093.front.sepia.ceph.com
2021-07-23T06:22:31.803 INFO:teuthology.kill:2021-07-23 06:22:31,803.803 INFO:teuthology.lock.ops:unlocked smithi063.front.sepia.ceph.com
2021-07-23T06:24:32.097 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-07-23T06:24:32.184 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-07-23T06:24:32.185 INFO:teuthology.nuke:Checking targets against current locks
2021-07-23T06:24:32.258 INFO:teuthology.nuke:Not nuking smithi063.front.sepia.ceph.com because description doesn't match
2021-07-23T06:24:32.259 INFO:teuthology.nuke:Not nuking smithi093.front.sepia.ceph.com because description doesn't match