2021-06-03T16:07:25.661 INFO:root:teuthology version: 1.1.0-f359b10d
2021-06-03T16:07:25.753 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi062.front.sepia.ceph.com' reimaging
2021-06-03T16:07:25.754 INFO:teuthology.lock.ops:Updating [ubuntu@smithi062.front.sepia.ceph.com]: reset os type and version on server
2021-06-03T16:07:25.754 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server
2021-06-03T16:07:25.769 INFO:teuthology.lock.ops:Node 'ubuntu@smithi062.front.sepia.ceph.com' reimaging is complete
2021-06-03T16:07:25.769 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi072.front.sepia.ceph.com' reimaging
2021-06-03T16:07:25.770 INFO:teuthology.lock.ops:Updating [ubuntu@smithi072.front.sepia.ceph.com]: reset os type and version on server
2021-06-03T16:07:25.770 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server
2021-06-03T16:07:25.799 INFO:teuthology.lock.ops:Node 'ubuntu@smithi072.front.sepia.ceph.com' reimaging is complete
2021-06-03T16:07:25.963 INFO:teuthology.provision.fog.smithi062:Scheduling deploy of centos 8.2
2021-06-03T16:07:25.986 INFO:teuthology.provision.fog.smithi072:Scheduling deploy of centos 8.2
2021-06-03T16:07:26.292 INFO:teuthology.orchestra.console:Power off smithi062
2021-06-03T16:07:26.328 INFO:teuthology.orchestra.console:Power off smithi072
2021-06-03T16:07:34.691 INFO:teuthology.orchestra.console:Power off for smithi072 completed
2021-06-03T16:07:34.792 INFO:teuthology.orchestra.console:Power on smithi072
2021-06-03T16:07:38.772 INFO:teuthology.orchestra.console:Power off for smithi062 completed
2021-06-03T16:07:38.874 INFO:teuthology.orchestra.console:Power on smithi062
2021-06-03T16:07:39.041 INFO:teuthology.orchestra.console:Power on for smithi072 completed
2021-06-03T16:07:39.142 INFO:teuthology.provision.fog.smithi072:Waiting for deploy to finish
2021-06-03T16:07:43.128 INFO:teuthology.orchestra.console:Power on for smithi062 completed
2021-06-03T16:07:43.229 INFO:teuthology.provision.fog.smithi062:Waiting for deploy to finish
2021-06-03T16:12:24.248 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-06-03T16:12:24.600 INFO:teuthology.provision.fog.smithi062:Node is ready
2021-06-03T16:12:24.615 INFO:teuthology.orchestra.run.smithi062.stdout:smithi062.front.sepia.ceph.com
2021-06-03T16:12:24.671 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62 smithi062.front.sepia.ceph.com smithi062
2021-06-03T16:12:24.991 INFO:teuthology.provision.fog.smithi062:Deploy complete!
2021-06-03T16:16:15.823 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-06-03T16:16:31.427 INFO:teuthology.provision.fog.smithi072:Node is ready
2021-06-03T16:16:31.442 INFO:teuthology.orchestra.run.smithi072.stdout:smithi072.front.sepia.ceph.com
2021-06-03T16:16:31.502 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72 smithi072.front.sepia.ceph.com smithi072
2021-06-03T16:16:31.949 INFO:teuthology.provision.fog.smithi072:Deploy complete!
2021-06-03T16:16:32.067 INFO:teuthology.lock.ops:Checking smithi062.front.sepia.ceph.com
2021-06-03T16:16:32.068 INFO:teuthology.lock.ops:New key found. Updating...
2021-06-03T16:16:32.082 INFO:teuthology.lock.ops:Checking smithi072.front.sepia.ceph.com
2021-06-03T16:16:32.083 INFO:teuthology.lock.ops:New key found. Updating...
2021-06-03T16:16:32.095 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: set os type and version on server
2021-06-03T16:16:32.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-06-03T16:16:32.312 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64
2021-06-03T16:16:32.368 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Linux"
2021-06-03T16:16:32.369 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="8 (Core)"
2021-06-03T16:16:32.369 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos"
2021-06-03T16:16:32.369 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora"
2021-06-03T16:16:32.369 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="8"
2021-06-03T16:16:32.370 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el8"
2021-06-03T16:16:32.370 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Linux 8 (Core)"
2021-06-03T16:16:32.370 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31"
2021-06-03T16:16:32.370 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-06-03T16:16:32.371 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://www.centos.org/"
2021-06-03T16:16:32.371 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-06-03T16:16:32.371 INFO:teuthology.orchestra.run.smithi062.stdout:
2021-06-03T16:16:32.371 INFO:teuthology.orchestra.run.smithi062.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-06-03T16:16:32.371 INFO:teuthology.orchestra.run.smithi062.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-06-03T16:16:32.372 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="centos"
2021-06-03T16:16:32.372 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8"
2021-06-03T16:16:32.372 INFO:teuthology.orchestra.run.smithi062.stdout:
2021-06-03T16:16:32.373 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server
2021-06-03T16:16:32.388 INFO:teuthology.lock.ops:Updating [smithi072.front.sepia.ceph.com]: set os type and version on server
2021-06-03T16:16:32.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-06-03T16:16:32.650 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64
2021-06-03T16:16:32.703 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Linux"
2021-06-03T16:16:32.704 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="8 (Core)"
2021-06-03T16:16:32.704 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos"
2021-06-03T16:16:32.704 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora"
2021-06-03T16:16:32.705 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="8"
2021-06-03T16:16:32.705 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el8"
2021-06-03T16:16:32.705 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Linux 8 (Core)"
2021-06-03T16:16:32.705 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31"
2021-06-03T16:16:32.705 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-06-03T16:16:32.706 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://www.centos.org/"
2021-06-03T16:16:32.706 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-06-03T16:16:32.706 INFO:teuthology.orchestra.run.smithi072.stdout:
2021-06-03T16:16:32.706 INFO:teuthology.orchestra.run.smithi072.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-06-03T16:16:32.707 INFO:teuthology.orchestra.run.smithi072.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-06-03T16:16:32.707 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="centos"
2021-06-03T16:16:32.707 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8"
2021-06-03T16:16:32.707 INFO:teuthology.orchestra.run.smithi072.stdout:
2021-06-03T16:16:32.708 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server
2021-06-03T16:16:32.783 INFO:teuthology.dispatcher.supervisor:Running job 6147264
2021-06-03T16:16:32.784 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-06-02_18:29:36-fs-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi/6147264 --name yuriw-2021-06-02_18:29:36-fs-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi --description fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/ffsb}} -- /home/teuthworker/archive/yuriw-2021-06-02_18:29:36-fs-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi/6147264/orig.config.yaml
2021-06-03T16:16:32.787 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-06-02_18:29:36-fs-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi/6147264
2021-06-03T16:16:32.788 INFO:teuthology.dispatcher.supervisor:Job PID: 21706
2021-06-03T16:16:32.788 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-06-03T19:10:45.418 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2021-06-02_18:29:36-fs-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-06-03T23:29:04.345 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2021-06-02_18:29:36-fs-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-06-04T04:17:25.411 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-06-04T04:17:25.700 INFO:teuthology.kill:Killing Pids: {21706}
2021-06-04T04:17:26.979 INFO:teuthology.kill:Nuking machines: ['smithi062', 'smithi072']
2021-06-04T04:17:28.121 INFO:teuthology.kill:2021-06-04 04:17:28,121.121 INFO:teuthology.nuke:targets:
2021-06-04T04:17:28.122 INFO:teuthology.kill: smithi062.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdAaJLSCO4+q8Jz4mdk1FmWXbEwH20y7qPC0mQC3P8z61hUTGPuKe1TBe139si4u7BkHs55WciRyottErKEnJhKx0FJsy2NN7s8scz25nT2+xoJcupjUqmsggErQBc5WIFHT6d+xoMskSJaH/W2xiWWlWDpKpYeaQRYf9A2cP96tpHwXUNSTzuTFcW3pFzaAQUhpDGaEDpy4bt0NJ/6y+UZTUItfpfQFT/WPCaQsi9kKVeKgtN2AFl8UrjqqANgxtEX277+mx9MBzh2aJkISbyq69r8hlMKriwSTmJOoJgrN6/zcdTV2ArC/cRTgXqIlJuJqXPxkrIDkNUwKLv+u6MhXy5y9xLFYvUn3C0ZBi6nxN4lJXdq8+v1+EOvA3tO4bCnul+YuL9bkITsvsEW+pvLhheIG6aUaod0Zqt3la2dQRUVr/dH2hj9djtb6HHbwy5ihaMu7KYmwiB8oq9UkUJ0aROY5a4xXYiWeHMvl1ZfHJueicmhgNamogPYjLyqkU=
2021-06-04T04:17:28.122 INFO:teuthology.kill: smithi072.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfREAjGD9Vz95yVmZCZMjlyDmslhnPbtS8laFWzs4La2cK/SlbyJ3+5QfzpLVGfEoARtM8XS/JKnL4u0pbOXKUxjw9C0MAKe8u3NOqWgSu+oIiY/X2hgxGLNWuLlc6AQmVmunOgyu4/+6Z2A6UxeLyuHVpDi4EtO/e7QAhV7Tp00FwW/aL9RkzvLCjvaVgMDWbMHZjKqLdb7lUwlApEL6S+jA41J/UGcdYzj73UiF/2gfF57r7xDxV4K1/CE2mQTkaUNLjhWqhQGbVx9k5UMY3fzQYmHmnf6ogFT0v4rcJRRBTNztz6JkYdwAM2jvj+RjR+En/Q4+wvw18bNDkMSPFRytE1rfaANcm+1Ytfy91gl+v84HmM3kk6LphkAU8FTZg7zBIKsP5xQFFm7B4dQzEVzMLkr6qWNfv2w7nVXssrnC9uh/AKaJQNnuemMbKZQP69rcLY0obOwnsdjuSygFLnl4aDqaupae8w8K/N8K3dZGzUS8sDbywC+0A6cCiQA8=
2021-06-04T04:17:28.123 INFO:teuthology.kill:2021-06-04 04:17:28,121.121 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-06-04T04:17:28.124 INFO:teuthology.kill:2021-06-04 04:17:28,124.124 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-06-04T04:17:28.192 INFO:teuthology.kill:2021-06-04 04:17:28,192.192 INFO:teuthology.orchestra.console:Power off smithi062
2021-06-04T04:17:28.212 INFO:teuthology.kill:2021-06-04 04:17:28,212.212 INFO:teuthology.orchestra.console:Power off smithi072
2021-06-04T04:17:36.580 INFO:teuthology.kill:2021-06-04 04:17:36,580.580 INFO:teuthology.orchestra.console:Power off for smithi072 completed
2021-06-04T04:17:40.687 INFO:teuthology.kill:2021-06-04 04:17:40,687.687 INFO:teuthology.orchestra.console:Power off for smithi062 completed
2021-06-04T04:17:40.933 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2021-06-04T04:17:40.933 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2021-06-04T04:17:40.934 INFO:teuthology.misc:Compressing logs...
2021-06-04T04:17:40.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-06-04T04:18:40.996 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
teuth_config.archive_base, job_config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
compress_logs(ctx, log_path)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run
transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-06-04T04:18:41.080 INFO:teuthology.kill:No teuthology processes running
2021-06-04T04:18:42.382 INFO:teuthology.kill:Nuking machines: ['smithi062', 'smithi072']
2021-06-04T04:18:43.403 INFO:teuthology.kill:2021-06-04 04:18:43,403.403 INFO:teuthology.nuke:targets:
2021-06-04T04:18:43.404 INFO:teuthology.kill: smithi062.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdAaJLSCO4+q8Jz4mdk1FmWXbEwH20y7qPC0mQC3P8z61hUTGPuKe1TBe139si4u7BkHs55WciRyottErKEnJhKx0FJsy2NN7s8scz25nT2+xoJcupjUqmsggErQBc5WIFHT6d+xoMskSJaH/W2xiWWlWDpKpYeaQRYf9A2cP96tpHwXUNSTzuTFcW3pFzaAQUhpDGaEDpy4bt0NJ/6y+UZTUItfpfQFT/WPCaQsi9kKVeKgtN2AFl8UrjqqANgxtEX277+mx9MBzh2aJkISbyq69r8hlMKriwSTmJOoJgrN6/zcdTV2ArC/cRTgXqIlJuJqXPxkrIDkNUwKLv+u6MhXy5y9xLFYvUn3C0ZBi6nxN4lJXdq8+v1+EOvA3tO4bCnul+YuL9bkITsvsEW+pvLhheIG6aUaod0Zqt3la2dQRUVr/dH2hj9djtb6HHbwy5ihaMu7KYmwiB8oq9UkUJ0aROY5a4xXYiWeHMvl1ZfHJueicmhgNamogPYjLyqkU=
2021-06-04T04:18:43.404 INFO:teuthology.kill: smithi072.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfREAjGD9Vz95yVmZCZMjlyDmslhnPbtS8laFWzs4La2cK/SlbyJ3+5QfzpLVGfEoARtM8XS/JKnL4u0pbOXKUxjw9C0MAKe8u3NOqWgSu+oIiY/X2hgxGLNWuLlc6AQmVmunOgyu4/+6Z2A6UxeLyuHVpDi4EtO/e7QAhV7Tp00FwW/aL9RkzvLCjvaVgMDWbMHZjKqLdb7lUwlApEL6S+jA41J/UGcdYzj73UiF/2gfF57r7xDxV4K1/CE2mQTkaUNLjhWqhQGbVx9k5UMY3fzQYmHmnf6ogFT0v4rcJRRBTNztz6JkYdwAM2jvj+RjR+En/Q4+wvw18bNDkMSPFRytE1rfaANcm+1Ytfy91gl+v84HmM3kk6LphkAU8FTZg7zBIKsP5xQFFm7B4dQzEVzMLkr6qWNfv2w7nVXssrnC9uh/AKaJQNnuemMbKZQP69rcLY0obOwnsdjuSygFLnl4aDqaupae8w8K/N8K3dZGzUS8sDbywC+0A6cCiQA8=
2021-06-04T04:18:43.424 INFO:teuthology.kill:2021-06-04 04:18:43,424.424 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-06-04T04:18:43.428 INFO:teuthology.kill:2021-06-04 04:18:43,427.427 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-06-04T04:18:43.476 INFO:teuthology.kill:2021-06-04 04:18:43,475.475 INFO:teuthology.orchestra.console:Power off smithi062
2021-06-04T04:18:43.489 INFO:teuthology.kill:2021-06-04 04:18:43,489.489 INFO:teuthology.orchestra.console:Power off smithi072
2021-06-04T04:18:43.616 INFO:teuthology.kill:2021-06-04 04:18:43,616.616 INFO:teuthology.orchestra.console:Power off for smithi062 completed
2021-06-04T04:18:43.629 INFO:teuthology.kill:2021-06-04 04:18:43,628.628 INFO:teuthology.orchestra.console:Power off for smithi072 completed
2021-06-04T04:18:43.769 INFO:teuthology.kill:2021-06-04 04:18:43,769.769 INFO:teuthology.lock.ops:unlocked smithi062.front.sepia.ceph.com
2021-06-04T04:18:43.781 INFO:teuthology.kill:2021-06-04 04:18:43,781.781 INFO:teuthology.lock.ops:unlocked smithi072.front.sepia.ceph.com
2021-06-04T04:20:44.089 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-06-04T04:20:44.203 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-06-04T04:20:44.204 INFO:teuthology.nuke:Checking targets against current locks
2021-06-04T04:20:44.274 INFO:teuthology.nuke:Not nuking smithi062.front.sepia.ceph.com because description doesn't match
2021-06-04T04:20:44.275 INFO:teuthology.nuke:Not nuking smithi072.front.sepia.ceph.com because description doesn't match