2021-05-26T05:57:19.705 INFO:root:teuthology version: 1.1.0-0c70841a 2021-05-26T05:57:19.794 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi032.front.sepia.ceph.com' reimaging 2021-05-26T05:57:19.795 INFO:teuthology.lock.ops:Updating [ubuntu@smithi032.front.sepia.ceph.com]: reset os type and version on server 2021-05-26T05:57:19.795 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2021-05-26T05:57:19.808 INFO:teuthology.lock.ops:Node 'ubuntu@smithi032.front.sepia.ceph.com' reimaging is complete 2021-05-26T05:57:19.809 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi078.front.sepia.ceph.com' reimaging 2021-05-26T05:57:19.809 INFO:teuthology.lock.ops:Updating [ubuntu@smithi078.front.sepia.ceph.com]: reset os type and version on server 2021-05-26T05:57:19.809 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2021-05-26T05:57:19.823 INFO:teuthology.lock.ops:Node 'ubuntu@smithi078.front.sepia.ceph.com' reimaging is complete 2021-05-26T05:57:20.015 INFO:teuthology.provision.fog.smithi078:Scheduling deploy of ubuntu 20.04 2021-05-26T05:57:20.018 INFO:teuthology.provision.fog.smithi032:Scheduling deploy of ubuntu 20.04 2021-05-26T05:57:20.372 INFO:teuthology.orchestra.console:Power off smithi032 2021-05-26T05:57:20.387 INFO:teuthology.orchestra.console:Power off smithi078 2021-05-26T05:57:28.744 INFO:teuthology.orchestra.console:Power off for smithi032 completed 2021-05-26T05:57:28.748 INFO:teuthology.orchestra.console:Power off for smithi078 completed 2021-05-26T05:57:28.846 INFO:teuthology.orchestra.console:Power on smithi032 2021-05-26T05:57:28.853 INFO:teuthology.orchestra.console:Power on smithi078 2021-05-26T05:57:37.213 INFO:teuthology.orchestra.console:Power on for smithi032 completed 2021-05-26T05:57:37.219 INFO:teuthology.orchestra.console:Power on for smithi078 completed 2021-05-26T05:57:37.315 INFO:teuthology.provision.fog.smithi032:Waiting for deploy to finish 2021-05-26T05:57:37.320 INFO:teuthology.provision.fog.smithi078:Waiting for deploy to finish 2021-05-26T06:02:15.191 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-26T06:02:15.658 INFO:teuthology.provision.fog.smithi032:Node is ready 2021-05-26T06:02:15.665 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032.front.sepia.ceph.com 2021-05-26T06:02:15.672 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32 smithi032.front.sepia.ceph.com smithi032 2021-05-26T06:02:15.874 INFO:teuthology.provision.fog.smithi032:Deploy complete! 2021-05-26T06:02:32.125 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-26T06:02:32.563 INFO:teuthology.provision.fog.smithi078:Node is ready 2021-05-26T06:02:32.570 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078.front.sepia.ceph.com 2021-05-26T06:02:32.577 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78 smithi078.front.sepia.ceph.com smithi078 2021-05-26T06:02:32.804 INFO:teuthology.provision.fog.smithi078:Deploy complete! 2021-05-26T06:02:32.953 INFO:teuthology.lock.ops:Checking smithi032.front.sepia.ceph.com 2021-05-26T06:02:32.954 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-26T06:02:32.969 INFO:teuthology.lock.ops:Checking smithi078.front.sepia.ceph.com 2021-05-26T06:02:32.970 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-26T06:02:32.985 INFO:teuthology.lock.ops:Updating [smithi032.front.sepia.ceph.com]: set os type and version on server 2021-05-26T06:02:32.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-26T06:02:33.250 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2021-05-26T06:02:33.300 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="Ubuntu" 2021-05-26T06:02:33.300 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-26T06:02:33.300 INFO:teuthology.orchestra.run.smithi032.stdout:ID=ubuntu 2021-05-26T06:02:33.301 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE=debian 2021-05-26T06:02:33.301 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-26T06:02:33.301 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="20.04" 2021-05-26T06:02:33.301 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-26T06:02:33.302 INFO:teuthology.orchestra.run.smithi032.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-26T06:02:33.302 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-26T06:02:33.302 INFO:teuthology.orchestra.run.smithi032.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-26T06:02:33.302 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_CODENAME=focal 2021-05-26T06:02:33.302 INFO:teuthology.orchestra.run.smithi032.stdout:UBUNTU_CODENAME=focal 2021-05-26T06:02:33.303 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2021-05-26T06:02:33.319 INFO:teuthology.lock.ops:Updating [smithi078.front.sepia.ceph.com]: set os type and version on server 2021-05-26T06:02:33.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-26T06:02:33.487 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2021-05-26T06:02:33.536 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="Ubuntu" 2021-05-26T06:02:33.536 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-26T06:02:33.536 INFO:teuthology.orchestra.run.smithi078.stdout:ID=ubuntu 2021-05-26T06:02:33.537 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE=debian 2021-05-26T06:02:33.537 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-26T06:02:33.537 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="20.04" 2021-05-26T06:02:33.537 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-26T06:02:33.538 INFO:teuthology.orchestra.run.smithi078.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-26T06:02:33.538 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-26T06:02:33.538 INFO:teuthology.orchestra.run.smithi078.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-26T06:02:33.538 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_CODENAME=focal 2021-05-26T06:02:33.539 INFO:teuthology.orchestra.run.smithi078.stdout:UBUNTU_CODENAME=focal 2021-05-26T06:02:33.539 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2021-05-26T06:02:33.611 INFO:teuthology.dispatcher.supervisor:Running job 6134856 2021-05-26T06:02:33.611 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-05-25_19:30:22-fs-wip-yuri4-testing-2021-05-25-0940-pacific-distro-basic-smithi/6134856 --name yuriw-2021-05-25_19:30:22-fs-wip-yuri4-testing-2021-05-25-0940-pacific-distro-basic-smithi --description fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/fragment} -- /home/teuthworker/archive/yuriw-2021-05-25_19:30:22-fs-wip-yuri4-testing-2021-05-25-0940-pacific-distro-basic-smithi/6134856/orig.config.yaml 2021-05-26T06:02:33.615 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-05-25_19:30:22-fs-wip-yuri4-testing-2021-05-25-0940-pacific-distro-basic-smithi/6134856 2021-05-26T06:02:33.616 INFO:teuthology.dispatcher.supervisor:Job PID: 32190 2021-05-26T06:02:33.616 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-26T18:03:22.475 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-26T18:03:22.680 INFO:teuthology.kill:Killing Pids: {32190} 2021-05-26T18:03:23.781 INFO:teuthology.kill:Nuking machines: ['smithi032', 'smithi078'] 2021-05-26T18:03:24.738 INFO:teuthology.kill:2021-05-26 18:03:24,738.738 INFO:teuthology.nuke:targets: 2021-05-26T18:03:24.739 INFO:teuthology.kill: smithi032.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBwGoOuN8GVoRV406hfWDANcGVStfO02epajgJYlIiivuRru41tqNiU5HRthQM1+P+zRYq4d4sq58rbBVc8cSK1Docv9V/WLQgWUJMPqQiQTIt0Kgfv88LpyCUWjwzgqXX8tpie7Wsk1OZG2GPl9AoYOS/NbahMb63a85ctSkyrO+gewJt+m1l1Jsza5nb7xX+YxswI1LhuRnqOi5Khr5uOZPayAwLi2ZGp+twtQ5HXlHx3y3advioYHrVfbLhFblXSKPpjEGQr+tcz80auO8NzyeQT3WXGNLi7pm9/YVvYsaAAHADVjV++1s9e6irrQ39CWFxt+FlmHRqkOMh9MEv3t4GsWBM9ZMCFLTg+6fD85eTPCC3vZaZ863HsbbnUSHkokOmGK/c/+9dcsgEy3hOx0k2IAoyVfs+BBoS6aJcTF/B+0Y+ZaViVhGKlW5JHh09kNihbZhCCpt2FrmgBuLO/AtbOwgdP4VoNEIFL9qPaeTH8KjEBE7iK8v57T6apR0= 2021-05-26T18:03:24.739 INFO:teuthology.kill: smithi078.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkr45bPnVmKOKpdE8qXD1wbTBOETGdEaQM2Cjsmf+FH1KxEqqxpdNRG7V1Srr7fTh+m+YUZprdzm9F+pDWnpghKIsf5fReEn+YU6quwioz4wO4RbLR+v6x0us25aAoVJzBeTjMeK0HXuL7+Mn4KztNHoA30Ed0UXw+6yLxJSMI4OqKpJXl1WBTuQeQr7DcbRH3eaBAIpdjGv4aRdZ2MMyBNbFnAKFCOIp27I86yBNtIQxt1UDh+joQgZL3ifItgnGMcinE2FhsXrwrEgj9OgZPAHc28/CDLKjwu8si+v7yFstR7QSvykblKfV1+haQkr+UQ/2aV0gY/q7TZRSYSjqf3T0jwGWkwQxwU2QdaDDoXsmwXWCPswHPLuIKUge4Zs5sJQd9CK8+WI2TyvwzXty8GxwAYjaecXr1OJD6nSCVMC2Ur3wabYwMH6m2o2FQgnwdYbYHpu7Zfpgt2ouSmpv4duBrfcVsy5MBTXZ8hOmaq6l23iuCrgap+hd0OImIkz0= 2021-05-26T18:03:24.740 INFO:teuthology.kill:2021-05-26 18:03:24,738.738 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T18:03:24.741 INFO:teuthology.kill:2021-05-26 18:03:24,741.741 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T18:03:24.802 INFO:teuthology.kill:2021-05-26 18:03:24,802.802 INFO:teuthology.orchestra.console:Power off smithi078 2021-05-26T18:03:24.817 INFO:teuthology.kill:2021-05-26 18:03:24,817.817 INFO:teuthology.orchestra.console:Power off smithi032 2021-05-26T18:03:33.177 INFO:teuthology.kill:2021-05-26 18:03:33,176.176 INFO:teuthology.orchestra.console:Power off for smithi032 completed 2021-05-26T18:03:37.283 INFO:teuthology.kill:2021-05-26 18:03:37,283.283 INFO:teuthology.orchestra.console:Power off for smithi078 completed 2021-05-26T18:03:37.510 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-05-26T18:03:37.511 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-05-26T18:03:37.511 INFO:teuthology.misc:Compressing logs... 2021-05-26T18:03:37.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-26T18:04:37.571 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-26T18:04:37.640 INFO:teuthology.kill:No teuthology processes running 2021-05-26T18:04:38.719 INFO:teuthology.kill:Nuking machines: ['smithi032', 'smithi078'] 2021-05-26T18:04:39.649 INFO:teuthology.kill:2021-05-26 18:04:39,649.649 INFO:teuthology.nuke:targets: 2021-05-26T18:04:39.650 INFO:teuthology.kill: smithi032.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBwGoOuN8GVoRV406hfWDANcGVStfO02epajgJYlIiivuRru41tqNiU5HRthQM1+P+zRYq4d4sq58rbBVc8cSK1Docv9V/WLQgWUJMPqQiQTIt0Kgfv88LpyCUWjwzgqXX8tpie7Wsk1OZG2GPl9AoYOS/NbahMb63a85ctSkyrO+gewJt+m1l1Jsza5nb7xX+YxswI1LhuRnqOi5Khr5uOZPayAwLi2ZGp+twtQ5HXlHx3y3advioYHrVfbLhFblXSKPpjEGQr+tcz80auO8NzyeQT3WXGNLi7pm9/YVvYsaAAHADVjV++1s9e6irrQ39CWFxt+FlmHRqkOMh9MEv3t4GsWBM9ZMCFLTg+6fD85eTPCC3vZaZ863HsbbnUSHkokOmGK/c/+9dcsgEy3hOx0k2IAoyVfs+BBoS6aJcTF/B+0Y+ZaViVhGKlW5JHh09kNihbZhCCpt2FrmgBuLO/AtbOwgdP4VoNEIFL9qPaeTH8KjEBE7iK8v57T6apR0= 2021-05-26T18:04:39.650 INFO:teuthology.kill: smithi078.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkr45bPnVmKOKpdE8qXD1wbTBOETGdEaQM2Cjsmf+FH1KxEqqxpdNRG7V1Srr7fTh+m+YUZprdzm9F+pDWnpghKIsf5fReEn+YU6quwioz4wO4RbLR+v6x0us25aAoVJzBeTjMeK0HXuL7+Mn4KztNHoA30Ed0UXw+6yLxJSMI4OqKpJXl1WBTuQeQr7DcbRH3eaBAIpdjGv4aRdZ2MMyBNbFnAKFCOIp27I86yBNtIQxt1UDh+joQgZL3ifItgnGMcinE2FhsXrwrEgj9OgZPAHc28/CDLKjwu8si+v7yFstR7QSvykblKfV1+haQkr+UQ/2aV0gY/q7TZRSYSjqf3T0jwGWkwQxwU2QdaDDoXsmwXWCPswHPLuIKUge4Zs5sJQd9CK8+WI2TyvwzXty8GxwAYjaecXr1OJD6nSCVMC2Ur3wabYwMH6m2o2FQgnwdYbYHpu7Zfpgt2ouSmpv4duBrfcVsy5MBTXZ8hOmaq6l23iuCrgap+hd0OImIkz0= 2021-05-26T18:04:39.673 INFO:teuthology.kill:2021-05-26 18:04:39,673.673 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T18:04:39.677 INFO:teuthology.kill:2021-05-26 18:04:39,676.676 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T18:04:39.743 INFO:teuthology.kill:2021-05-26 18:04:39,743.743 INFO:teuthology.orchestra.console:Power off smithi078 2021-05-26T18:04:39.761 INFO:teuthology.kill:2021-05-26 18:04:39,760.760 INFO:teuthology.orchestra.console:Power off smithi032 2021-05-26T18:04:39.897 INFO:teuthology.kill:2021-05-26 18:04:39,897.897 INFO:teuthology.orchestra.console:Power off for smithi078 completed 2021-05-26T18:04:39.913 INFO:teuthology.kill:2021-05-26 18:04:39,912.912 INFO:teuthology.orchestra.console:Power off for smithi032 completed 2021-05-26T18:04:40.040 INFO:teuthology.kill:2021-05-26 18:04:40,040.040 INFO:teuthology.lock.ops:unlocked smithi078.front.sepia.ceph.com 2021-05-26T18:04:40.056 INFO:teuthology.kill:2021-05-26 18:04:40,055.055 INFO:teuthology.lock.ops:unlocked smithi032.front.sepia.ceph.com 2021-05-26T18:06:40.375 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-05-26T18:06:40.465 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-05-26T18:06:40.466 INFO:teuthology.nuke:Checking targets against current locks 2021-05-26T18:06:40.541 INFO:teuthology.nuke:Not nuking smithi032.front.sepia.ceph.com because description doesn't match 2021-05-26T18:06:40.541 INFO:teuthology.nuke:Not nuking smithi078.front.sepia.ceph.com because description doesn't match