2021-11-17T18:16:53.533 INFO:root:teuthology version: 1.1.0-d4737010 2021-11-17T18:16:53.655 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba035.front.sepia.ceph.com' reimaging 2021-11-17T18:16:53.656 INFO:teuthology.lock.ops:Updating [ubuntu@gibba035.front.sepia.ceph.com]: reset os type and version on server 2021-11-17T18:16:53.656 INFO:teuthology.lock.ops:Updating gibba035.front.sepia.ceph.com on lock server 2021-11-17T18:16:53.675 INFO:teuthology.lock.ops:Node 'ubuntu@gibba035.front.sepia.ceph.com' reimaging is complete 2021-11-17T18:16:53.675 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba045.front.sepia.ceph.com' reimaging 2021-11-17T18:16:53.676 INFO:teuthology.lock.ops:Updating [ubuntu@gibba045.front.sepia.ceph.com]: reset os type and version on server 2021-11-17T18:16:53.676 INFO:teuthology.lock.ops:Updating gibba045.front.sepia.ceph.com on lock server 2021-11-17T18:16:53.694 INFO:teuthology.lock.ops:Node 'ubuntu@gibba045.front.sepia.ceph.com' reimaging is complete 2021-11-17T18:16:53.872 INFO:teuthology.provision.fog.gibba035:Scheduling deploy of centos 8.stream 2021-11-17T18:16:53.908 INFO:teuthology.provision.fog.gibba045:Scheduling deploy of centos 8.stream 2021-11-17T18:16:54.183 INFO:teuthology.orchestra.console:Power off gibba035 2021-11-17T18:16:54.223 INFO:teuthology.orchestra.console:Power off gibba045 2021-11-17T18:17:02.565 INFO:teuthology.orchestra.console:Power off for gibba035 completed 2021-11-17T18:17:02.605 INFO:teuthology.orchestra.console:Power off for gibba045 completed 2021-11-17T18:17:02.667 INFO:teuthology.orchestra.console:Power on gibba035 2021-11-17T18:17:02.706 INFO:teuthology.orchestra.console:Power on gibba045 2021-11-17T18:17:11.042 INFO:teuthology.orchestra.console:Power on for gibba035 completed 2021-11-17T18:17:11.080 INFO:teuthology.orchestra.console:Power on for gibba045 completed 2021-11-17T18:17:11.144 INFO:teuthology.provision.fog.gibba035:Waiting for deploy to finish 2021-11-17T18:17:11.181 INFO:teuthology.provision.fog.gibba045:Waiting for deploy to finish 2021-11-17T18:21:07.937 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-17T18:21:08.038 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-17T18:21:18.547 INFO:teuthology.provision.fog.gibba035:Node is ready 2021-11-17T18:21:18.561 INFO:teuthology.orchestra.run.gibba035.stdout:gibba035.front.sepia.ceph.com 2021-11-17T18:21:18.615 INFO:teuthology.orchestra.run.gibba035.stdout:172.21.2.135 gibba035.front.sepia.ceph.com gibba035 2021-11-17T18:21:18.956 INFO:teuthology.provision.fog.gibba035:Deploy complete! 2021-11-17T18:21:23.354 INFO:teuthology.provision.fog.gibba045:Node is ready 2021-11-17T18:21:23.370 INFO:teuthology.orchestra.run.gibba045.stdout:gibba045.front.sepia.ceph.com 2021-11-17T18:21:23.437 INFO:teuthology.orchestra.run.gibba045.stdout:172.21.2.145 gibba045.front.sepia.ceph.com gibba045 2021-11-17T18:21:23.803 INFO:teuthology.provision.fog.gibba045:Deploy complete! 2021-11-17T18:21:24.246 INFO:teuthology.lock.ops:Checking gibba035.front.sepia.ceph.com 2021-11-17T18:21:24.247 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-17T18:21:24.268 INFO:teuthology.lock.ops:Checking gibba045.front.sepia.ceph.com 2021-11-17T18:21:24.268 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-17T18:21:24.286 INFO:teuthology.lock.ops:Updating [gibba035.front.sepia.ceph.com]: set os type and version on server 2021-11-17T18:21:24.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@gibba035.front.sepia.ceph.com' 2021-11-17T18:21:24.495 INFO:teuthology.orchestra.run.gibba035.stdout:x86_64 2021-11-17T18:21:24.549 INFO:teuthology.orchestra.run.gibba035.stdout:NAME="CentOS Stream" 2021-11-17T18:21:24.549 INFO:teuthology.orchestra.run.gibba035.stdout:VERSION="8" 2021-11-17T18:21:24.549 INFO:teuthology.orchestra.run.gibba035.stdout:ID="centos" 2021-11-17T18:21:24.550 INFO:teuthology.orchestra.run.gibba035.stdout:ID_LIKE="rhel fedora" 2021-11-17T18:21:24.550 INFO:teuthology.orchestra.run.gibba035.stdout:VERSION_ID="8" 2021-11-17T18:21:24.550 INFO:teuthology.orchestra.run.gibba035.stdout:PLATFORM_ID="platform:el8" 2021-11-17T18:21:24.550 INFO:teuthology.orchestra.run.gibba035.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-17T18:21:24.551 INFO:teuthology.orchestra.run.gibba035.stdout:ANSI_COLOR="0;31" 2021-11-17T18:21:24.551 INFO:teuthology.orchestra.run.gibba035.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-17T18:21:24.551 INFO:teuthology.orchestra.run.gibba035.stdout:HOME_URL="https://centos.org/" 2021-11-17T18:21:24.551 INFO:teuthology.orchestra.run.gibba035.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-17T18:21:24.552 INFO:teuthology.orchestra.run.gibba035.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-17T18:21:24.552 INFO:teuthology.orchestra.run.gibba035.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-17T18:21:24.553 INFO:teuthology.lock.ops:Updating gibba035.front.sepia.ceph.com on lock server 2021-11-17T18:21:24.573 INFO:teuthology.lock.ops:Updating [gibba045.front.sepia.ceph.com]: set os type and version on server 2021-11-17T18:21:24.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@gibba045.front.sepia.ceph.com' 2021-11-17T18:21:24.787 INFO:teuthology.orchestra.run.gibba045.stdout:x86_64 2021-11-17T18:21:24.840 INFO:teuthology.orchestra.run.gibba045.stdout:NAME="CentOS Stream" 2021-11-17T18:21:24.840 INFO:teuthology.orchestra.run.gibba045.stdout:VERSION="8" 2021-11-17T18:21:24.841 INFO:teuthology.orchestra.run.gibba045.stdout:ID="centos" 2021-11-17T18:21:24.841 INFO:teuthology.orchestra.run.gibba045.stdout:ID_LIKE="rhel fedora" 2021-11-17T18:21:24.841 INFO:teuthology.orchestra.run.gibba045.stdout:VERSION_ID="8" 2021-11-17T18:21:24.841 INFO:teuthology.orchestra.run.gibba045.stdout:PLATFORM_ID="platform:el8" 2021-11-17T18:21:24.841 INFO:teuthology.orchestra.run.gibba045.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-17T18:21:24.842 INFO:teuthology.orchestra.run.gibba045.stdout:ANSI_COLOR="0;31" 2021-11-17T18:21:24.842 INFO:teuthology.orchestra.run.gibba045.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-17T18:21:24.842 INFO:teuthology.orchestra.run.gibba045.stdout:HOME_URL="https://centos.org/" 2021-11-17T18:21:24.842 INFO:teuthology.orchestra.run.gibba045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-17T18:21:24.843 INFO:teuthology.orchestra.run.gibba045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-17T18:21:24.843 INFO:teuthology.orchestra.run.gibba045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-17T18:21:24.843 INFO:teuthology.lock.ops:Updating gibba045.front.sepia.ceph.com on lock server 2021-11-17T18:21:24.955 INFO:teuthology.dispatcher.supervisor:Running job 6502943 2021-11-17T18:21:24.955 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-11-14_07:01:03-rados-master-distro-default-gibba/6502943 --name teuthology-2021-11-14_07:01:03-rados-master-distro-default-gibba --description rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/teuthology-2021-11-14_07:01:03-rados-master-distro-default-gibba/6502943/orig.config.yaml 2021-11-17T18:21:24.958 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-11-14_07:01:03-rados-master-distro-default-gibba/6502943 2021-11-17T18:21:24.959 INFO:teuthology.dispatcher.supervisor:Job PID: 20771 2021-11-17T18:21:24.960 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-11-18T06:22:19.881 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-11-18T06:22:20.130 INFO:teuthology.kill:Killing Pids: {20771} 2021-11-18T06:22:22.096 INFO:teuthology.task.internal:roles: ubuntu@gibba035.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-11-18T06:22:22.099 INFO:teuthology.task.internal:roles: ubuntu@gibba045.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-11-18T06:22:22.099 INFO:teuthology.misc:Compressing logs... 2021-11-18T06:22:22.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@gibba035.front.sepia.ceph.com' 2021-11-18T06:22:22.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@gibba045.front.sepia.ceph.com' 2021-11-18T06:22:22.309 INFO:teuthology.orchestra.run.gibba035.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-11-18T06:22:22.486 INFO:teuthology.orchestra.run.gibba045.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-11-18T06:22:22.523 INFO:teuthology.orchestra.run.gibba035.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-11-18T06:22:22.523 INFO:teuthology.orchestra.run.gibba035.stderr:tar: Error is not recoverable: exiting now 2021-11-18T06:22:22.560 INFO:teuthology.orchestra.run.gibba045.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-11-18T06:22:22.561 INFO:teuthology.orchestra.run.gibba045.stderr:tar: Error is not recoverable: exiting now 2021-11-18T06:22:22.562 INFO:teuthology.misc:Compressing logs... 2021-11-18T06:22:22.992 INFO:teuthology.misc:Compressing logs... 2021-11-18T06:22:25.664 INFO:teuthology.orchestra.run.gibba035.stderr:gzip: /var/log/ceph/7460b50c-47d4-11ec-8c2c-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2021-11-18T06:22:26.537 INFO:teuthology.orchestra.run.gibba045.stderr:gzip: /var/log/ceph/7460b50c-47d4-11ec-8c2c-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2021-11-18T06:22:29.254 INFO:teuthology.orchestra.run.gibba035.stderr:gzip: /var/log/ceph/7460b50c-47d4-11ec-8c2c-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2021-11-18T06:22:30.278 INFO:teuthology.orchestra.run.gibba045.stderr:gzip: /var/log/ceph/7460b50c-47d4-11ec-8c2c-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2021-11-18T06:22:32.376 INFO:teuthology.orchestra.run.gibba045.stderr:gzip: /var/log/ceph/7460b50c-47d4-11ec-8c2c-001a4aab830c/ceph-mon.gibba045.log: file size changed while zipping 2021-11-18T06:22:33.029 INFO:teuthology.orchestra.run.gibba035.stderr:gzip: /var/log/ceph/7460b50c-47d4-11ec-8c2c-001a4aab830c/ceph-mon.gibba035.log: file size changed while zipping 2021-11-18T06:22:36.674 INFO:teuthology.orchestra.run.gibba045.stderr:gzip: /var/log/ceph/7460b50c-47d4-11ec-8c2c-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2021-11-18T06:22:36.800 INFO:teuthology.orchestra.run.gibba035.stderr:gzip: /var/log/ceph/7460b50c-47d4-11ec-8c2c-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2021-11-18T06:22:38.316 INFO:teuthology.orchestra.run.gibba045.stderr:gzip: /var/log/ceph/7460b50c-47d4-11ec-8c2c-001a4aab830c/ceph-mgr.gibba045.bbpsdd.log: file size changed while zipping 2021-11-18T06:22:40.718 INFO:teuthology.orchestra.run.gibba035.stderr:gzip: /var/log/ceph/7460b50c-47d4-11ec-8c2c-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2021-11-18T06:22:42.635 INFO:teuthology.orchestra.run.gibba045.stderr:gzip: /var/log/ceph/7460b50c-47d4-11ec-8c2c-001a4aab830c/ceph-osd.7.log: file size changed while zipping 2021-11-18T06:23:26.676 INFO:teuthology.kill:No teuthology processes running 2021-11-18T06:23:27.919 INFO:teuthology.kill:Nuking machines: ['gibba035', 'gibba045'] 2021-11-18T06:23:28.918 INFO:teuthology.kill:2021-11-18 06:23:28,917.917 INFO:teuthology.nuke:targets: 2021-11-18T06:23:28.918 INFO:teuthology.kill: gibba035.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLPNzd0HOuaft5SEhDm6PFbbhB5EDpwL44+F6HFNID3zU+39ElVW/C/Te71RrlgqfBlO0lWxbpmQinW/W1RoZ/5gdntmeSxV6UONyaooDq9fhu7zRG6dCfEAbomRvcq+w+z3vXqly+ShO5pkUDWm6PwUNuDpM4hMYlf0OUNWK1s59Obm3IkBQSkLb8e7S+AAxlWshHMQMgEVDKswlGoN0uTnsKCDx7AjKuyD9VmhBDeIfuqjI9MZVT+dgWdx2mP4CFjCx80TNic4WCs7mERDGfvJdwkf2ZIPmqR5mJ4ZM2mQWpuabyBM9r2EjMutHYxWi6cwR0ZrDsPvL4oSKXg8Aa+8qWdRVV1PqiH4HDMFX20XZ5GuUNfnsfhGTlii+bz4sF7moMPZ8uYPdEvyUs3kWr1y8XlIodB57bTCOOhluJP2D4t/sq6ItkJMZ4uN5CSOFx7I3kuqVzb9A/Xjf1puyoNnXLvkCIMpeXFN9Y23aVvcmp3oeEXDiDyKknxmb8S6U= 2021-11-18T06:23:28.919 INFO:teuthology.kill: gibba045.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN8PMgFyzLND6cdDlTd7jxUWJJkLEJTkFMoxswFPgFJWiZFCNFu1fVkfGT83UiZOns9Qsg1cEooIZknvwsLawTUdwiKR01aPg/OuVjEszDLMpPTd//4UCk4hbwPSUYbs9kgUCdS9EYZaJbEZ83+WZNIsfNxFFhB/T5QuuAE2MDMo3SLxsLEGc1ML74XqCoLP/C5RmZpMGZAmJYvDpr20RLT646C6/OAXFKJ75lIIbXpnxZf2ZfPhBki2/9/U1p77ygrRqx3b8eBR01fcaiAXusyS2eQGYiR+eNBZrvjYh/rKNfcdxcBlJ5QxOic0pQJ6tqXCjMUk1uOZ8Si6V9E+f4SCh7b9U5zHKcELzbkJS+WkMyMIJ8Yvn5Eu2ZD0ghNmhEk+ZuKnTqx1jpc1kdy9gTxJnH3TFOwoXDXhhW6+RwrN471AxPE3sOY1w1tFkbW8tQ45Pn1t42Ct3y1PbsmB6EWrPqZMOt7+QmkzOOuZSsHn11LZJhzKch6J6vupj08B8= 2021-11-18T06:23:28.948 INFO:teuthology.kill:2021-11-18 06:23:28,947.947 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-18T06:23:28.951 INFO:teuthology.kill:2021-11-18 06:23:28,951.951 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-18T06:23:29.022 INFO:teuthology.kill:2021-11-18 06:23:29,022.022 INFO:teuthology.orchestra.console:Power off gibba035 2021-11-18T06:23:29.037 INFO:teuthology.kill:2021-11-18 06:23:29,036.036 INFO:teuthology.orchestra.console:Power off gibba045 2021-11-18T06:23:37.431 INFO:teuthology.kill:2021-11-18 06:23:37,430.430 INFO:teuthology.orchestra.console:Power off for gibba045 completed 2021-11-18T06:23:37.496 INFO:teuthology.kill:2021-11-18 06:23:37,496.496 INFO:teuthology.orchestra.console:Power off for gibba035 completed 2021-11-18T06:23:37.589 INFO:teuthology.kill:2021-11-18 06:23:37,588.588 INFO:teuthology.lock.ops:unlocked: gibba045.front.sepia.ceph.com 2021-11-18T06:23:37.651 INFO:teuthology.kill:2021-11-18 06:23:37,651.651 INFO:teuthology.lock.ops:unlocked: gibba035.front.sepia.ceph.com 2021-11-18T06:25:38.031 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-11-18T06:25:38.141 WARNING:teuthology.dispatcher.supervisor:Was going to unlock gibba035 but it was locked by another job: /home/teuthworker/archive/teuthology-2021-11-18_05:00:03-smoke-master-distro-default-gibba/6511321 2021-11-18T06:25:38.141 WARNING:teuthology.dispatcher.supervisor:Was going to unlock gibba045 but it was locked by another job: /home/teuthworker/archive/teuthology-2021-11-18_05:00:03-smoke-master-distro-default-gibba/6511321