2021-04-28T21:06:44.668 INFO:root:teuthology version: 1.1.0-2713a3cd
2021-04-28T21:06:44.819 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi076.front.sepia.ceph.com' reimaging
2021-04-28T21:06:44.819 INFO:teuthology.lock.ops:Updating [ubuntu@smithi076.front.sepia.ceph.com]: reset os type and version on server
2021-04-28T21:06:44.819 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server
2021-04-28T21:06:44.835 INFO:teuthology.lock.ops:Node 'ubuntu@smithi076.front.sepia.ceph.com' reimaging is complete
2021-04-28T21:06:44.835 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi140.front.sepia.ceph.com' reimaging
2021-04-28T21:06:44.836 INFO:teuthology.lock.ops:Updating [ubuntu@smithi140.front.sepia.ceph.com]: reset os type and version on server
2021-04-28T21:06:44.836 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2021-04-28T21:06:44.853 INFO:teuthology.lock.ops:Node 'ubuntu@smithi140.front.sepia.ceph.com' reimaging is complete
2021-04-28T21:06:45.069 INFO:teuthology.provision.fog.smithi076:Scheduling deploy of centos 8.2
2021-04-28T21:06:45.076 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of centos 8.2
2021-04-28T21:06:45.421 INFO:teuthology.orchestra.console:Power off smithi076
2021-04-28T21:06:45.496 INFO:teuthology.orchestra.console:Power off smithi140
2021-04-28T21:06:53.861 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2021-04-28T21:06:53.962 INFO:teuthology.orchestra.console:Power on smithi140
2021-04-28T21:06:57.909 INFO:teuthology.orchestra.console:Power off for smithi076 completed
2021-04-28T21:06:58.012 INFO:teuthology.orchestra.console:Power on smithi076
2021-04-28T21:07:02.262 INFO:teuthology.orchestra.console:Power on for smithi076 completed
2021-04-28T21:07:02.357 INFO:teuthology.orchestra.console:Power on for smithi140 completed
2021-04-28T21:07:02.364 INFO:teuthology.provision.fog.smithi076:Waiting for deploy to finish
2021-04-28T21:07:02.458 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish
2021-04-28T21:11:58.487 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-04-28T21:11:58.551 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-04-28T21:11:58.792 INFO:teuthology.provision.fog.smithi076:Node is ready
2021-04-28T21:11:58.806 INFO:teuthology.orchestra.run.smithi076.stdout:smithi076.front.sepia.ceph.com
2021-04-28T21:11:58.820 INFO:teuthology.provision.fog.smithi140:Node is ready
2021-04-28T21:11:58.835 INFO:teuthology.orchestra.run.smithi140.stdout:smithi140.front.sepia.ceph.com
2021-04-28T21:11:58.860 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76 smithi076.front.sepia.ceph.com smithi076
2021-04-28T21:11:58.890 INFO:teuthology.orchestra.run.smithi140.stdout:172.21.15.140 smithi140.front.sepia.ceph.com smithi140
2021-04-28T21:11:59.182 INFO:teuthology.provision.fog.smithi076:Deploy complete!
2021-04-28T21:11:59.204 INFO:teuthology.provision.fog.smithi140:Deploy complete!
2021-04-28T21:12:29.034 INFO:teuthology.lock.ops:Checking smithi076.front.sepia.ceph.com
2021-04-28T21:12:29.035 INFO:teuthology.lock.ops:New key found. Updating...
2021-04-28T21:12:29.920 INFO:teuthology.lock.ops:Checking smithi140.front.sepia.ceph.com
2021-04-28T21:12:29.920 INFO:teuthology.lock.ops:New key found. Updating...
2021-04-28T21:12:42.499 INFO:teuthology.lock.ops:Updating [smithi076.front.sepia.ceph.com]: set os type and version on server
2021-04-28T21:12:42.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-28T21:12:42.781 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64
2021-04-28T21:12:42.835 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Linux"
2021-04-28T21:12:42.836 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="8 (Core)"
2021-04-28T21:12:42.836 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos"
2021-04-28T21:12:42.836 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora"
2021-04-28T21:12:42.836 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="8"
2021-04-28T21:12:42.836 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el8"
2021-04-28T21:12:42.836 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Linux 8 (Core)"
2021-04-28T21:12:42.836 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31"
2021-04-28T21:12:42.836 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-04-28T21:12:42.836 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://www.centos.org/"
2021-04-28T21:12:42.836 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-04-28T21:12:42.836 INFO:teuthology.orchestra.run.smithi076.stdout:
2021-04-28T21:12:42.836 INFO:teuthology.orchestra.run.smithi076.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-04-28T21:12:42.837 INFO:teuthology.orchestra.run.smithi076.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-04-28T21:12:42.837 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="centos"
2021-04-28T21:12:42.837 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8"
2021-04-28T21:12:42.837 INFO:teuthology.orchestra.run.smithi076.stdout:
2021-04-28T21:12:42.838 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server
2021-04-28T21:12:56.197 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: set os type and version on server
2021-04-28T21:12:56.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-28T21:12:56.467 INFO:teuthology.orchestra.run.smithi140.stdout:x86_64
2021-04-28T21:12:56.521 INFO:teuthology.orchestra.run.smithi140.stdout:NAME="CentOS Linux"
2021-04-28T21:12:56.521 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION="8 (Core)"
2021-04-28T21:12:56.521 INFO:teuthology.orchestra.run.smithi140.stdout:ID="centos"
2021-04-28T21:12:56.521 INFO:teuthology.orchestra.run.smithi140.stdout:ID_LIKE="rhel fedora"
2021-04-28T21:12:56.521 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION_ID="8"
2021-04-28T21:12:56.521 INFO:teuthology.orchestra.run.smithi140.stdout:PLATFORM_ID="platform:el8"
2021-04-28T21:12:56.522 INFO:teuthology.orchestra.run.smithi140.stdout:PRETTY_NAME="CentOS Linux 8 (Core)"
2021-04-28T21:12:56.522 INFO:teuthology.orchestra.run.smithi140.stdout:ANSI_COLOR="0;31"
2021-04-28T21:12:56.522 INFO:teuthology.orchestra.run.smithi140.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-04-28T21:12:56.522 INFO:teuthology.orchestra.run.smithi140.stdout:HOME_URL="https://www.centos.org/"
2021-04-28T21:12:56.522 INFO:teuthology.orchestra.run.smithi140.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-04-28T21:12:56.522 INFO:teuthology.orchestra.run.smithi140.stdout:
2021-04-28T21:12:56.522 INFO:teuthology.orchestra.run.smithi140.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-04-28T21:12:56.522 INFO:teuthology.orchestra.run.smithi140.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-04-28T21:12:56.522 INFO:teuthology.orchestra.run.smithi140.stdout:REDHAT_SUPPORT_PRODUCT="centos"
2021-04-28T21:12:56.522 INFO:teuthology.orchestra.run.smithi140.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8"
2021-04-28T21:12:56.522 INFO:teuthology.orchestra.run.smithi140.stdout:
2021-04-28T21:12:56.523 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2021-04-28T21:12:57.310 INFO:teuthology.dispatcher.supervisor:Running job 6080166
2021-04-28T21:12:57.311 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-04-28_19:13:18-rados-pacific-distro-basic-smithi/6080166 --name yuriw-2021-04-28_19:13:18-rados-pacific-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects} -- /home/teuthworker/archive/yuriw-2021-04-28_19:13:18-rados-pacific-distro-basic-smithi/6080166/orig.config.yaml
2021-04-28T21:12:57.315 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-04-28_19:13:18-rados-pacific-distro-basic-smithi/6080166
2021-04-28T21:12:57.315 INFO:teuthology.dispatcher.supervisor:Job PID: 29974
2021-04-28T21:12:57.315 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-04-28T21:45:44.872 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2021-04-28_19:13:18-rados-pacific-distro-basic-smithi/jobs/ failed with status 504:
504 Gateway Time-out
504 Gateway Time-out
nginx/1.10.3 (Ubuntu)
2021-04-28T21:46:45.915 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2021-04-28_19:13:18-rados-pacific-distro-basic-smithi/jobs/ failed with status 504:
504 Gateway Time-out
504 Gateway Time-out
nginx/1.10.3 (Ubuntu)
2021-04-29T03:13:32.324 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2021-04-28_19:13:18-rados-pacific-distro-basic-smithi/jobs/ failed with status 504:
504 Gateway Time-out
504 Gateway Time-out
nginx/1.10.3 (Ubuntu)
2021-04-29T09:13:07.524 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-04-29T09:13:07.770 INFO:teuthology.kill:Killing Pids: {29974}
2021-04-29T09:13:08.883 INFO:teuthology.kill:Nuking machines: ['smithi076', 'smithi140']
2021-04-29T09:13:09.848 INFO:teuthology.kill:2021-04-29 09:13:09,847.847 INFO:teuthology.nuke:targets:
2021-04-29T09:13:09.848 INFO:teuthology.kill: smithi076.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAVuklCaieWsdBlTK09PKfaKHhuWis7l1D51Ov2AxPW8xksnGUxGa/kLYYg2GdknXs5ETdDYiYDy6+UMwVpzKQ5s+u7cX52qQmBShNbMYhxvAc9EDjpSsbI2Yw2Xfr6QcCOglTS6RhP6ddU1cYXZ42WEPT7tukaFEZ9yx1FYjqWVpgvnzp7tphtH2cPZJ7zPQ/WfnLzSNmf++Qrn4QxxgVYYA1aRShuFHml6mjDdbqB4JbsxDBGC79s62msGPvc80ZfnLOkZpC1ae/1R7QjNXTDp929mnoBZ+pF9Bt+SAMFpiDU09XmqGcpahAfuyDqZOglhCFdwCFnF+VGHU9xy83bk6fE1QVD+ah06gstBqhO/EnaITWUM/Xgu8zXQ5AiTVrir7r84c+h26TWPeNAcDvPUWquCIPVh0q36OPEPyHMjTijn+mcrFGHNSp4DhjkacwKM9oineqHE9d9BCh4nMEvvclUf1V56rjSOMemPQ+VOwONiHduugiaqcw1Fsrd/k=
2021-04-29T09:13:09.848 INFO:teuthology.kill: smithi140.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDC0z7fUATGag4uxiyKWm0mfMlA3EPpsdwpO3upcv7o+6t1I+/5QsY+7efib/2cbVoG9scGI7+6lVhBUK28tRJMQglOlUbmr5fQbao/3u07qCM5GRCnCaknPXzkXGbI6gp6j04wY6DJvtHEGYPVgXkMPJP7Za/Wf2OLhEZI/rhAA10Zp+ypKJPn8uRjQxfnQoaLNinj7mnIGqZ7DU3UCr8vrKAFDVTjzLBa3fpDCPw+CbV3tGjAx/uKtzPtUNF3ocrqjUeM61rAvS2PgPgE6cxk6XaiOgAzkDFAJB8MxYxgYEidOJrK4tF/SwL1iSI2/cWTzcDntia93rbahaQA6uB595ixxL3FnYdH8IE3uFCNH3tXBFa5GHQc6cdc7e2MvR8Q03Yq99dutzJCEa1v4NvJ58llhpsGoFVYu6+w3ZIYZ4y30dMfkKrbdavL+3yXbm2wnjrHH8zMdoCRbrxfyJSxfOELb42eq5sn4mp39MRz+iD8I5Fby6Qxob9STFyssC0=
2021-04-29T09:13:09.848 INFO:teuthology.kill:2021-04-29 09:13:09,848.848 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-29T09:13:09.850 INFO:teuthology.kill:2021-04-29 09:13:09,850.850 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-29T09:13:09.926 INFO:teuthology.kill:2021-04-29 09:13:09,925.925 INFO:teuthology.orchestra.console:Power off smithi076
2021-04-29T09:13:09.946 INFO:teuthology.kill:2021-04-29 09:13:09,945.945 INFO:teuthology.orchestra.console:Power off smithi140
2021-04-29T09:13:18.332 INFO:teuthology.kill:2021-04-29 09:13:18,332.332 INFO:teuthology.orchestra.console:Power off for smithi076 completed
2021-04-29T09:13:18.337 INFO:teuthology.kill:2021-04-29 09:13:18,337.337 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2021-04-29T09:13:18.618 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2021-04-29T09:13:18.618 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2021-04-29T09:13:18.618 INFO:teuthology.misc:Compressing logs...
2021-04-29T09:13:18.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-29T09:14:18.682 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
teuth_config.archive_base, job_config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
compress_logs(ctx, log_path)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/misc.py", line 1378, in compress_logs
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/remote.py", line 215, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 434, in run
transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-04-29T09:14:18.749 INFO:teuthology.kill:No teuthology processes running
2021-04-29T09:14:19.818 INFO:teuthology.kill:Nuking machines: ['smithi076', 'smithi140']
2021-04-29T09:14:20.746 INFO:teuthology.kill:2021-04-29 09:14:20,745.745 INFO:teuthology.nuke:targets:
2021-04-29T09:14:20.746 INFO:teuthology.kill: smithi076.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAVuklCaieWsdBlTK09PKfaKHhuWis7l1D51Ov2AxPW8xksnGUxGa/kLYYg2GdknXs5ETdDYiYDy6+UMwVpzKQ5s+u7cX52qQmBShNbMYhxvAc9EDjpSsbI2Yw2Xfr6QcCOglTS6RhP6ddU1cYXZ42WEPT7tukaFEZ9yx1FYjqWVpgvnzp7tphtH2cPZJ7zPQ/WfnLzSNmf++Qrn4QxxgVYYA1aRShuFHml6mjDdbqB4JbsxDBGC79s62msGPvc80ZfnLOkZpC1ae/1R7QjNXTDp929mnoBZ+pF9Bt+SAMFpiDU09XmqGcpahAfuyDqZOglhCFdwCFnF+VGHU9xy83bk6fE1QVD+ah06gstBqhO/EnaITWUM/Xgu8zXQ5AiTVrir7r84c+h26TWPeNAcDvPUWquCIPVh0q36OPEPyHMjTijn+mcrFGHNSp4DhjkacwKM9oineqHE9d9BCh4nMEvvclUf1V56rjSOMemPQ+VOwONiHduugiaqcw1Fsrd/k=
2021-04-29T09:14:20.746 INFO:teuthology.kill: smithi140.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDC0z7fUATGag4uxiyKWm0mfMlA3EPpsdwpO3upcv7o+6t1I+/5QsY+7efib/2cbVoG9scGI7+6lVhBUK28tRJMQglOlUbmr5fQbao/3u07qCM5GRCnCaknPXzkXGbI6gp6j04wY6DJvtHEGYPVgXkMPJP7Za/Wf2OLhEZI/rhAA10Zp+ypKJPn8uRjQxfnQoaLNinj7mnIGqZ7DU3UCr8vrKAFDVTjzLBa3fpDCPw+CbV3tGjAx/uKtzPtUNF3ocrqjUeM61rAvS2PgPgE6cxk6XaiOgAzkDFAJB8MxYxgYEidOJrK4tF/SwL1iSI2/cWTzcDntia93rbahaQA6uB595ixxL3FnYdH8IE3uFCNH3tXBFa5GHQc6cdc7e2MvR8Q03Yq99dutzJCEa1v4NvJ58llhpsGoFVYu6+w3ZIYZ4y30dMfkKrbdavL+3yXbm2wnjrHH8zMdoCRbrxfyJSxfOELb42eq5sn4mp39MRz+iD8I5Fby6Qxob9STFyssC0=
2021-04-29T09:14:20.772 INFO:teuthology.kill:2021-04-29 09:14:20,772.772 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-29T09:14:20.785 INFO:teuthology.kill:2021-04-29 09:14:20,785.785 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-29T09:14:20.855 INFO:teuthology.kill:2021-04-29 09:14:20,854.854 INFO:teuthology.orchestra.console:Power off smithi076
2021-04-29T09:14:20.868 INFO:teuthology.kill:2021-04-29 09:14:20,867.867 INFO:teuthology.orchestra.console:Power off smithi140
2021-04-29T09:14:20.995 INFO:teuthology.kill:2021-04-29 09:14:20,995.995 INFO:teuthology.orchestra.console:Power off for smithi076 completed
2021-04-29T09:14:21.009 INFO:teuthology.kill:2021-04-29 09:14:21,009.009 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2021-04-29T09:14:21.147 INFO:teuthology.kill:2021-04-29 09:14:21,146.146 INFO:teuthology.lock.ops:unlocked smithi076.front.sepia.ceph.com
2021-04-29T09:14:21.162 INFO:teuthology.kill:2021-04-29 09:14:21,162.162 INFO:teuthology.lock.ops:unlocked smithi140.front.sepia.ceph.com
2021-04-29T09:16:21.462 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-04-29T09:16:21.555 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-04-29T09:16:21.556 INFO:teuthology.nuke:Checking targets against current locks
2021-04-29T09:16:21.626 INFO:teuthology.nuke:Not nuking smithi076.front.sepia.ceph.com because description doesn't match
2021-04-29T09:16:21.626 INFO:teuthology.nuke:Not nuking smithi140.front.sepia.ceph.com because description doesn't match