2021-06-03T12:18:52.190 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-03T12:18:52.288 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi163.front.sepia.ceph.com' reimaging 2021-06-03T12:18:52.289 INFO:teuthology.lock.ops:Updating [ubuntu@smithi163.front.sepia.ceph.com]: reset os type and version on server 2021-06-03T12:18:52.290 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2021-06-03T12:18:52.305 INFO:teuthology.lock.ops:Node 'ubuntu@smithi163.front.sepia.ceph.com' reimaging is complete 2021-06-03T12:18:52.305 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi180.front.sepia.ceph.com' reimaging 2021-06-03T12:18:52.306 INFO:teuthology.lock.ops:Updating [ubuntu@smithi180.front.sepia.ceph.com]: reset os type and version on server 2021-06-03T12:18:52.306 INFO:teuthology.lock.ops:Updating smithi180.front.sepia.ceph.com on lock server 2021-06-03T12:18:52.323 INFO:teuthology.lock.ops:Node 'ubuntu@smithi180.front.sepia.ceph.com' reimaging is complete 2021-06-03T12:18:52.510 INFO:teuthology.provision.fog.smithi163:Scheduling deploy of centos 8.2 2021-06-03T12:18:52.524 INFO:teuthology.provision.fog.smithi180:Scheduling deploy of centos 8.2 2021-06-03T12:18:52.990 INFO:teuthology.orchestra.console:Power off smithi163 2021-06-03T12:18:53.030 INFO:teuthology.orchestra.console:Power off smithi180 2021-06-03T12:19:01.392 INFO:teuthology.orchestra.console:Power off for smithi180 completed 2021-06-03T12:19:01.494 INFO:teuthology.orchestra.console:Power on smithi180 2021-06-03T12:19:05.485 INFO:teuthology.orchestra.console:Power off for smithi163 completed 2021-06-03T12:19:05.586 INFO:teuthology.orchestra.console:Power on smithi163 2021-06-03T12:19:09.841 INFO:teuthology.orchestra.console:Power on for smithi163 completed 2021-06-03T12:19:09.855 INFO:teuthology.orchestra.console:Power on for smithi180 completed 2021-06-03T12:19:09.943 INFO:teuthology.provision.fog.smithi163:Waiting for deploy to finish 2021-06-03T12:19:09.956 INFO:teuthology.provision.fog.smithi180:Waiting for deploy to finish 2021-06-03T12:23:07.756 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-03T12:23:27.911 INFO:teuthology.provision.fog.smithi180:Node is ready 2021-06-03T12:23:27.932 INFO:teuthology.orchestra.run.smithi180.stdout:smithi180.front.sepia.ceph.com 2021-06-03T12:23:27.995 INFO:teuthology.orchestra.run.smithi180.stdout:172.21.15.180 smithi180.front.sepia.ceph.com smithi180 2021-06-03T12:23:28.394 INFO:teuthology.provision.fog.smithi180:Deploy complete! 2021-06-03T12:24:30.513 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-03T12:24:30.811 INFO:teuthology.provision.fog.smithi163:Node is ready 2021-06-03T12:24:30.826 INFO:teuthology.orchestra.run.smithi163.stdout:smithi163.front.sepia.ceph.com 2021-06-03T12:24:30.881 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.15.163 smithi163.front.sepia.ceph.com smithi163 2021-06-03T12:24:31.198 INFO:teuthology.provision.fog.smithi163:Deploy complete! 2021-06-03T12:24:31.323 INFO:teuthology.lock.ops:Checking smithi163.front.sepia.ceph.com 2021-06-03T12:24:31.324 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-03T12:24:31.340 INFO:teuthology.lock.ops:Checking smithi180.front.sepia.ceph.com 2021-06-03T12:24:31.340 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-03T12:24:31.353 INFO:teuthology.lock.ops:Updating [smithi163.front.sepia.ceph.com]: set os type and version on server 2021-06-03T12:24:31.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-03T12:24:31.561 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2021-06-03T12:24:31.614 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Linux" 2021-06-03T12:24:31.615 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="8 (Core)" 2021-06-03T12:24:31.615 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2021-06-03T12:24:31.615 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2021-06-03T12:24:31.615 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="8" 2021-06-03T12:24:31.616 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el8" 2021-06-03T12:24:31.617 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-06-03T12:24:31.617 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2021-06-03T12:24:31.617 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-03T12:24:31.617 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://www.centos.org/" 2021-06-03T12:24:31.619 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-03T12:24:31.619 INFO:teuthology.orchestra.run.smithi163.stdout: 2021-06-03T12:24:31.620 INFO:teuthology.orchestra.run.smithi163.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-03T12:24:31.620 INFO:teuthology.orchestra.run.smithi163.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-03T12:24:31.620 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-06-03T12:24:31.621 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-06-03T12:24:31.622 INFO:teuthology.orchestra.run.smithi163.stdout: 2021-06-03T12:24:31.625 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2021-06-03T12:24:31.642 INFO:teuthology.lock.ops:Updating [smithi180.front.sepia.ceph.com]: set os type and version on server 2021-06-03T12:24:31.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-03T12:24:31.916 INFO:teuthology.orchestra.run.smithi180.stdout:x86_64 2021-06-03T12:24:31.970 INFO:teuthology.orchestra.run.smithi180.stdout:NAME="CentOS Linux" 2021-06-03T12:24:31.970 INFO:teuthology.orchestra.run.smithi180.stdout:VERSION="8 (Core)" 2021-06-03T12:24:31.970 INFO:teuthology.orchestra.run.smithi180.stdout:ID="centos" 2021-06-03T12:24:31.971 INFO:teuthology.orchestra.run.smithi180.stdout:ID_LIKE="rhel fedora" 2021-06-03T12:24:31.971 INFO:teuthology.orchestra.run.smithi180.stdout:VERSION_ID="8" 2021-06-03T12:24:31.971 INFO:teuthology.orchestra.run.smithi180.stdout:PLATFORM_ID="platform:el8" 2021-06-03T12:24:31.971 INFO:teuthology.orchestra.run.smithi180.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-06-03T12:24:31.972 INFO:teuthology.orchestra.run.smithi180.stdout:ANSI_COLOR="0;31" 2021-06-03T12:24:31.972 INFO:teuthology.orchestra.run.smithi180.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-03T12:24:31.972 INFO:teuthology.orchestra.run.smithi180.stdout:HOME_URL="https://www.centos.org/" 2021-06-03T12:24:31.972 INFO:teuthology.orchestra.run.smithi180.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-03T12:24:31.973 INFO:teuthology.orchestra.run.smithi180.stdout: 2021-06-03T12:24:31.973 INFO:teuthology.orchestra.run.smithi180.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-03T12:24:31.973 INFO:teuthology.orchestra.run.smithi180.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-03T12:24:31.973 INFO:teuthology.orchestra.run.smithi180.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-06-03T12:24:31.973 INFO:teuthology.orchestra.run.smithi180.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-06-03T12:24:31.974 INFO:teuthology.orchestra.run.smithi180.stdout: 2021-06-03T12:24:31.974 INFO:teuthology.lock.ops:Updating smithi180.front.sepia.ceph.com on lock server 2021-06-03T12:24:32.074 INFO:teuthology.dispatcher.supervisor:Running job 6146958 2021-06-03T12:24:32.074 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-06-02_18:29:36-fs-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi/6146958 --name yuriw-2021-06-02_18:29:36-fs-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi --description fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc} -- /home/teuthworker/archive/yuriw-2021-06-02_18:29:36-fs-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi/6146958/orig.config.yaml 2021-06-03T12:24:32.077 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-06-02_18:29:36-fs-wip-yuri3-testing-2021-06-02-0826-pacific-distro-basic-smithi/6146958 2021-06-03T12:24:32.078 INFO:teuthology.dispatcher.supervisor:Job PID: 20443 2021-06-03T12:24:32.078 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-06-04T00:25:21.327 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-06-04T00:25:22.138 INFO:teuthology.kill:Killing Pids: {20443} 2021-06-04T00:25:23.213 INFO:teuthology.kill:Nuking machines: ['smithi163', 'smithi180'] 2021-06-04T00:25:24.166 INFO:teuthology.kill:2021-06-04 00:25:24,165.165 INFO:teuthology.nuke:targets: 2021-06-04T00:25:24.167 INFO:teuthology.kill: smithi163.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFOvR+D9RFIAMJiV7ZXamdU2hm0j1CjbhibU9B5Xe6nGJS27ycevkuxOFcjJiumJP8EMWJ7hmcOKQBAXe20x+cqUBH7/daRIO+KHCKQu4LkBM41WB/jD1zYfePrsZTeAAZN1lq0cmf4YwmZQEmzBKavQ5f+QXhLwaGmcr7bY9OcRFg5AYfCiFpaW16NisJ53SvvWPz9EJkZbdNOIBHrtiRvSd0kK04KpoDf7lFxLHN4EXMwZ3JOOHdEHp1z+Ae/+W3G7iSIrVGZJHWH7QHq5L3IlPKRKcEtIuQxvehSLhazDd9m/a+d5ZPls/kPSxzH/TCfzdR9EEyKGN4W3opsbNd8wU2IOVX8AwY4Othbgs9zK5rEsnLjT3Jq1755APx+BXFhPHIew4YGP8ZZDAptBU9pe7bjbyY/LeFIX2+C27L63YleB4/71Xn+YxRSsUlXWqNgEonANQv10BSFRNGWIL2CI9d04QNrL9LCl924HAu+pZW0gfcULeju97rssm6jPc= 2021-06-04T00:25:24.168 INFO:teuthology.kill: smithi180.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9mZhjdsrEkI1nz1ZBowxnxYEvQxcy2KNNKK2l7Y81CXT4lIsfl0pfzaHXgV9Lh+Cycnb0WzA5kb2sqdYBqRlCvZ/79nQmSD88cQPw1HypLij200iAS6px+v8N9q9l3NDwiOm/IwiK8LmczM86RC8jubiV5zNfp3QQ8QuTcX6/lSKPTvW75/05I+eMyOJnzO48OypUmHX/wDs0tzfxGsqDStjuC40/HHFdzf3g0EObs3dZJhNpbGi8Of0XnUhu7EExahrz3WnUHbakDZ2vQhIIbPzfR+XQR1oRhajwt7FwF7sgirGBvykvS8sFqkfV6AmwB9AgefnxQodxMxb367pBz8xkALGzvYcEMLtXQ0R5Mxd0pMYRbo+y4z/1MA/Bja5tP2TrJcA1m4jhTR10DkEQk70985ya/uC9RVD8UR2kSXF8MRW3xS2/pikDVZKm2jSOABwtZrbr/Om2p/uorD3fnPyhJR9JiPZKn+5+c3BCwZLmPOcc2uTodY4il55wdy8= 2021-06-04T00:25:24.168 INFO:teuthology.kill:2021-06-04 00:25:24,166.166 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-04T00:25:24.169 INFO:teuthology.kill:2021-06-04 00:25:24,168.168 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-04T00:25:24.229 INFO:teuthology.kill:2021-06-04 00:25:24,229.229 INFO:teuthology.orchestra.console:Power off smithi163 2021-06-04T00:25:24.243 INFO:teuthology.kill:2021-06-04 00:25:24,243.243 INFO:teuthology.orchestra.console:Power off smithi180 2021-06-04T00:25:32.601 INFO:teuthology.kill:2021-06-04 00:25:32,601.601 INFO:teuthology.orchestra.console:Power off for smithi163 completed 2021-06-04T00:25:32.604 INFO:teuthology.kill:2021-06-04 00:25:32,604.604 INFO:teuthology.orchestra.console:Power off for smithi180 completed 2021-06-04T00:25:32.841 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-06-04T00:25:32.842 INFO:teuthology.task.internal:roles: ubuntu@smithi180.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-06-04T00:25:32.842 INFO:teuthology.misc:Compressing logs... 2021-06-04T00:25:32.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-04T00:26:07.439 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-06-04T00:26:07.508 INFO:teuthology.kill:No teuthology processes running 2021-06-04T00:26:08.673 INFO:teuthology.kill:Nuking machines: ['smithi163', 'smithi180'] 2021-06-04T00:26:09.683 INFO:teuthology.kill:2021-06-04 00:26:09,683.683 INFO:teuthology.nuke:targets: 2021-06-04T00:26:09.684 INFO:teuthology.kill: smithi163.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFOvR+D9RFIAMJiV7ZXamdU2hm0j1CjbhibU9B5Xe6nGJS27ycevkuxOFcjJiumJP8EMWJ7hmcOKQBAXe20x+cqUBH7/daRIO+KHCKQu4LkBM41WB/jD1zYfePrsZTeAAZN1lq0cmf4YwmZQEmzBKavQ5f+QXhLwaGmcr7bY9OcRFg5AYfCiFpaW16NisJ53SvvWPz9EJkZbdNOIBHrtiRvSd0kK04KpoDf7lFxLHN4EXMwZ3JOOHdEHp1z+Ae/+W3G7iSIrVGZJHWH7QHq5L3IlPKRKcEtIuQxvehSLhazDd9m/a+d5ZPls/kPSxzH/TCfzdR9EEyKGN4W3opsbNd8wU2IOVX8AwY4Othbgs9zK5rEsnLjT3Jq1755APx+BXFhPHIew4YGP8ZZDAptBU9pe7bjbyY/LeFIX2+C27L63YleB4/71Xn+YxRSsUlXWqNgEonANQv10BSFRNGWIL2CI9d04QNrL9LCl924HAu+pZW0gfcULeju97rssm6jPc= 2021-06-04T00:26:09.685 INFO:teuthology.kill: smithi180.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9mZhjdsrEkI1nz1ZBowxnxYEvQxcy2KNNKK2l7Y81CXT4lIsfl0pfzaHXgV9Lh+Cycnb0WzA5kb2sqdYBqRlCvZ/79nQmSD88cQPw1HypLij200iAS6px+v8N9q9l3NDwiOm/IwiK8LmczM86RC8jubiV5zNfp3QQ8QuTcX6/lSKPTvW75/05I+eMyOJnzO48OypUmHX/wDs0tzfxGsqDStjuC40/HHFdzf3g0EObs3dZJhNpbGi8Of0XnUhu7EExahrz3WnUHbakDZ2vQhIIbPzfR+XQR1oRhajwt7FwF7sgirGBvykvS8sFqkfV6AmwB9AgefnxQodxMxb367pBz8xkALGzvYcEMLtXQ0R5Mxd0pMYRbo+y4z/1MA/Bja5tP2TrJcA1m4jhTR10DkEQk70985ya/uC9RVD8UR2kSXF8MRW3xS2/pikDVZKm2jSOABwtZrbr/Om2p/uorD3fnPyhJR9JiPZKn+5+c3BCwZLmPOcc2uTodY4il55wdy8= 2021-06-04T00:26:09.704 INFO:teuthology.kill:2021-06-04 00:26:09,704.704 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-04T00:26:09.708 INFO:teuthology.kill:2021-06-04 00:26:09,707.707 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-04T00:26:09.773 INFO:teuthology.kill:2021-06-04 00:26:09,772.772 INFO:teuthology.orchestra.console:Power off smithi180 2021-06-04T00:26:09.790 INFO:teuthology.kill:2021-06-04 00:26:09,790.790 INFO:teuthology.orchestra.console:Power off smithi163 2021-06-04T00:26:09.927 INFO:teuthology.kill:2021-06-04 00:26:09,927.927 INFO:teuthology.orchestra.console:Power off for smithi180 completed 2021-06-04T00:26:09.937 INFO:teuthology.kill:2021-06-04 00:26:09,937.937 INFO:teuthology.orchestra.console:Power off for smithi163 completed 2021-06-04T00:26:10.068 INFO:teuthology.kill:2021-06-04 00:26:10,068.068 INFO:teuthology.lock.ops:unlocked smithi180.front.sepia.ceph.com 2021-06-04T00:26:10.079 INFO:teuthology.kill:2021-06-04 00:26:10,079.079 INFO:teuthology.lock.ops:unlocked smithi163.front.sepia.ceph.com 2021-06-04T00:28:10.402 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-06-04T00:28:10.511 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-06-04T00:28:10.511 INFO:teuthology.nuke:Checking targets against current locks 2021-06-04T00:28:10.585 INFO:teuthology.nuke:Not nuking smithi180.front.sepia.ceph.com because description doesn't match 2021-06-04T00:28:10.585 INFO:teuthology.nuke:Not nuking smithi163.front.sepia.ceph.com because description doesn't match