2022-01-14T06:16:21.070 INFO:root:teuthology version: 1.1.0-6fc20113 2022-01-14T06:16:21.197 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi046.front.sepia.ceph.com' reimaging 2022-01-14T06:16:21.198 INFO:teuthology.lock.ops:Updating [ubuntu@smithi046.front.sepia.ceph.com]: reset os type and version on server 2022-01-14T06:16:21.199 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2022-01-14T06:16:21.217 INFO:teuthology.lock.ops:Node 'ubuntu@smithi046.front.sepia.ceph.com' reimaging is complete 2022-01-14T06:16:21.218 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi132.front.sepia.ceph.com' reimaging 2022-01-14T06:16:21.218 INFO:teuthology.lock.ops:Updating [ubuntu@smithi132.front.sepia.ceph.com]: reset os type and version on server 2022-01-14T06:16:21.218 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2022-01-14T06:16:21.236 INFO:teuthology.lock.ops:Node 'ubuntu@smithi132.front.sepia.ceph.com' reimaging is complete 2022-01-14T06:16:21.487 INFO:teuthology.provision.fog.smithi132:Scheduling deploy of centos 8.3 2022-01-14T06:16:21.526 INFO:teuthology.provision.fog.smithi046:Scheduling deploy of centos 8.3 2022-01-14T06:16:21.863 INFO:teuthology.orchestra.console:Power off smithi132 2022-01-14T06:16:21.929 INFO:teuthology.orchestra.console:Power off smithi046 2022-01-14T06:16:22.070 INFO:teuthology.orchestra.console:Power off for smithi046 completed 2022-01-14T06:16:22.171 INFO:teuthology.orchestra.console:Power on smithi046 2022-01-14T06:16:30.534 INFO:teuthology.orchestra.console:Power on for smithi046 completed 2022-01-14T06:16:30.635 INFO:teuthology.provision.fog.smithi046:Waiting for deploy to finish 2022-01-14T06:16:34.347 INFO:teuthology.orchestra.console:Power off for smithi132 completed 2022-01-14T06:16:34.448 INFO:teuthology.orchestra.console:Power on smithi132 2022-01-14T06:16:38.701 INFO:teuthology.orchestra.console:Power on for smithi132 completed 2022-01-14T06:16:38.803 INFO:teuthology.provision.fog.smithi132:Waiting for deploy to finish 2022-01-14T06:21:20.289 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-14T06:21:20.591 INFO:teuthology.provision.fog.smithi046:Node is ready 2022-01-14T06:21:20.648 INFO:teuthology.orchestra.run.smithi046.stdout:smithi046.front.sepia.ceph.com 2022-01-14T06:21:20.705 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46 smithi046.front.sepia.ceph.com smithi046 2022-01-14T06:21:21.050 INFO:teuthology.provision.fog.smithi046:Deploy complete! 2022-01-14T06:21:28.354 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-14T06:21:28.653 INFO:teuthology.provision.fog.smithi132:Node is ready 2022-01-14T06:21:28.667 INFO:teuthology.orchestra.run.smithi132.stdout:smithi132.front.sepia.ceph.com 2022-01-14T06:21:28.722 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.132 smithi132.front.sepia.ceph.com smithi132 2022-01-14T06:21:29.036 INFO:teuthology.provision.fog.smithi132:Deploy complete! 2022-01-14T06:21:29.180 INFO:teuthology.lock.ops:Checking smithi046.front.sepia.ceph.com 2022-01-14T06:21:29.181 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-14T06:21:29.200 INFO:teuthology.lock.ops:Checking smithi132.front.sepia.ceph.com 2022-01-14T06:21:29.200 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-14T06:21:29.218 INFO:teuthology.lock.ops:Updating [smithi046.front.sepia.ceph.com]: set os type and version on server 2022-01-14T06:21:29.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2022-01-14T06:21:29.497 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2022-01-14T06:21:29.553 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Linux" 2022-01-14T06:21:29.553 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="8" 2022-01-14T06:21:29.553 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2022-01-14T06:21:29.554 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2022-01-14T06:21:29.554 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="8" 2022-01-14T06:21:29.554 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el8" 2022-01-14T06:21:29.554 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Linux 8" 2022-01-14T06:21:29.555 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2022-01-14T06:21:29.555 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-14T06:21:29.555 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://centos.org/" 2022-01-14T06:21:29.555 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2022-01-14T06:21:29.556 INFO:teuthology.orchestra.run.smithi046.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2022-01-14T06:21:29.556 INFO:teuthology.orchestra.run.smithi046.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2022-01-14T06:21:29.557 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2022-01-14T06:21:29.576 INFO:teuthology.lock.ops:Updating [smithi132.front.sepia.ceph.com]: set os type and version on server 2022-01-14T06:21:29.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2022-01-14T06:21:29.817 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2022-01-14T06:21:29.871 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="CentOS Linux" 2022-01-14T06:21:29.871 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="8" 2022-01-14T06:21:29.871 INFO:teuthology.orchestra.run.smithi132.stdout:ID="centos" 2022-01-14T06:21:29.871 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="rhel fedora" 2022-01-14T06:21:29.872 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="8" 2022-01-14T06:21:29.872 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el8" 2022-01-14T06:21:29.872 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="CentOS Linux 8" 2022-01-14T06:21:29.872 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2022-01-14T06:21:29.872 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-14T06:21:29.873 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://centos.org/" 2022-01-14T06:21:29.873 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2022-01-14T06:21:29.873 INFO:teuthology.orchestra.run.smithi132.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2022-01-14T06:21:29.873 INFO:teuthology.orchestra.run.smithi132.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2022-01-14T06:21:29.874 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2022-01-14T06:21:29.961 INFO:teuthology.dispatcher.supervisor:Running job 6615161 2022-01-14T06:21:29.961 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_6fc2011361437a9dfe4e45b50de224392eed8abc/virtualenv/bin/teuthology -v --owner scheduled_gabrioux@teuthology --archive /home/teuthworker/archive/gabrioux-2022-01-13_23:13:28-rados:cephadm-wip-guits-testing-2022-01-13-001-pacific-distro-default-smithi/6615161 --name gabrioux-2022-01-13_23:13:28-rados:cephadm-wip-guits-testing-2022-01-13-001-pacific-distro-default-smithi --description rados:cephadm/mgr-nfs-upgrade/{0-distro/centos_8.3_container_tools_3.0 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/gabrioux-2022-01-13_23:13:28-rados:cephadm-wip-guits-testing-2022-01-13-001-pacific-distro-default-smithi/6615161/orig.config.yaml 2022-01-14T06:21:29.964 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/gabrioux-2022-01-13_23:13:28-rados:cephadm-wip-guits-testing-2022-01-13-001-pacific-distro-default-smithi/6615161 2022-01-14T06:21:29.965 INFO:teuthology.dispatcher.supervisor:Job PID: 3550 2022-01-14T06:21:29.965 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-01-14T12:51:59.575 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-01-14T12:51:59.787 INFO:teuthology.kill:Killing Pids: {3550} 2022-01-14T12:52:00.977 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-01-14T12:52:00.978 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-01-14T12:52:00.978 INFO:teuthology.misc:Compressing logs... 2022-01-14T12:52:00.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2022-01-14T12:52:01.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2022-01-14T12:52:01.230 INFO:teuthology.orchestra.run.smithi046.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2022-01-14T12:52:01.432 INFO:teuthology.orchestra.run.smithi132.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2022-01-14T12:52:01.467 INFO:teuthology.orchestra.run.smithi046.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2022-01-14T12:52:01.467 INFO:teuthology.orchestra.run.smithi046.stderr:tar: Error is not recoverable: exiting now 2022-01-14T12:52:01.507 INFO:teuthology.orchestra.run.smithi132.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2022-01-14T12:52:01.508 INFO:teuthology.orchestra.run.smithi132.stderr:tar: Error is not recoverable: exiting now 2022-01-14T12:52:01.509 INFO:teuthology.misc:Compressing logs... 2022-01-14T12:52:01.815 INFO:teuthology.misc:Compressing logs... 2022-01-14T12:52:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:gzip: /var/log/ceph/8b9f67ac-7503-11ec-8c32-001a4aab830c/ceph-mon.smithi046.log: file size changed while zipping 2022-01-14T12:52:30.856 INFO:teuthology.orchestra.run.smithi132.stderr:gzip: /var/log/ceph/8b9f67ac-7503-11ec-8c32-001a4aab830c/ceph-osd.7.log: file size changed while zipping 2022-01-14T12:52:40.905 INFO:teuthology.orchestra.run.smithi046.stderr:gzip: /var/log/ceph/8b9f67ac-7503-11ec-8c32-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-01-14T12:52:45.972 INFO:teuthology.orchestra.run.smithi046.stderr:gzip: /var/log/ceph/8b9f67ac-7503-11ec-8c32-001a4aab830c/ceph-mgr.smithi046.qwqbsa.log: file size changed while zipping 2022-01-14T12:52:55.013 INFO:teuthology.orchestra.run.smithi132.stderr:gzip: /var/log/ceph/8b9f67ac-7503-11ec-8c32-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-01-14T12:53:08.828 INFO:teuthology.orchestra.run.smithi046.stderr:gzip: /var/log/ceph/8b9f67ac-7503-11ec-8c32-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-01-14T12:53:15.673 INFO:teuthology.orchestra.run.smithi132.stderr:gzip: /var/log/ceph/8b9f67ac-7503-11ec-8c32-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-01-14T12:53:19.069 INFO:teuthology.orchestra.run.smithi132.stderr:gzip: /var/log/ceph/8b9f67ac-7503-11ec-8c32-001a4aab830c/ceph-mon.smithi132.log: file size changed while zipping 2022-01-14T12:53:25.022 INFO:teuthology.orchestra.run.smithi046.stderr:gzip: /var/log/ceph/8b9f67ac-7503-11ec-8c32-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-01-14T12:53:39.315 INFO:teuthology.orchestra.run.smithi132.stderr:gzip: /var/log/ceph/8b9f67ac-7503-11ec-8c32-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2022-01-14T12:53:51.321 INFO:teuthology.orchestra.run.smithi046.stderr:gzip: /var/log/ceph/8b9f67ac-7503-11ec-8c32-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-01-14T12:54:13.783 INFO:teuthology.orchestra.run.smithi046.stderr:tar: .: file changed as we read it 2022-01-14T12:54:33.083 INFO:teuthology.kill:No teuthology processes running 2022-01-14T12:54:34.314 INFO:teuthology.kill:Nuking machines: ['smithi046', 'smithi132'] 2022-01-14T12:54:35.311 INFO:teuthology.kill:2022-01-14 12:54:35,311.311 INFO:teuthology.nuke:targets: 2022-01-14T12:54:35.312 INFO:teuthology.kill: smithi046.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwlvouLw0OZt3J5nFQrHyf6Xnvv7F/udEqmP92aQ6TqGwhCFqBlMtAImpNGBKSRXeHsv/kcytBD3Jalol9q5qEadwq6fgUYUAfVbu03gqqnQ61Rjejiji1GXPeP4IdKK2KMC7vfKTZz+XfQPSIlBF1lxzWlyzQzqSLQk0Ry6umFHIfrO/VL5HCXon2dbi/5Q916RcyqkJow5jF3njx1+c2voiph/zUnumvAGODuVXtahEcUgSbaVBm/D4smE0JnDrwgGWg12ItUH3INqJoq09QwuTogyc+AuDYZMyjHnd51iOW8XipTNAG4haGuM3nIfidEFQ5b2IE8Q2cD4PmSTGukQ+BHvRh/Sxn3AbBAGl8Zq1kLRrB8jYemsaaKqYJqxXI5Uo/umzzX3wjetp8yQIB8P3A9lx5O/lMFDjRqJPiN7iAbVyYpyywTz9GVdUufX9a8Ptb8USoFQoTd5fWVh/e5BQO7vzbZqlNTha+ZRXvCkFQV3lV+y73hmTOP5FF+gM= 2022-01-14T12:54:35.313 INFO:teuthology.kill: smithi132.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDe9b6nfFoAnY00sfomTtgI1wIXuluGI1oSl3FcfgVFvbyqIWfVXZUTyzjK4IzG5KLawmHf51dhVjLpjM7Zyfl2dA5cPhn86738J4kThP1qWk7YS1daePG6RwSJJ1xRz8vWjPWzXLcGzgvd041Q1xvLroxg+tHciHPHueWwuf9ay98cStW6seFppg+vq0/iLFU/2vAUDXqMbzw6EX8S58agRcQEWVlthYpqpmJOamiZ+zC2eGGDlHE9k4SwshLVfuue+wQmkmoTFfoJ6lcV1eCsqkkbeOhtL0pGbBSqMcNAoq6iYAJVYxEYRACE9Z+7jAN8M6cTeGsF2zSyJw6DOf2S67pLz9pdA+2sSXCZtguAjbErWHldFsuuXo6aR5ZgXMoTlXJBzQwif+ACnwGiAzmSCAQ9+zFqBpIMSdz/3uyqiPSFCg9gmAGc02uFWYXbFhJRcoO/2Kmng7UHCKojuRCo0T4RclZwKhubB1njnfcLEqhXlKYHTIGihDnflbivEB8= 2022-01-14T12:54:35.337 INFO:teuthology.kill:2022-01-14 12:54:35,337.337 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-14T12:54:35.340 INFO:teuthology.kill:2022-01-14 12:54:35,340.340 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-14T12:54:35.413 INFO:teuthology.kill:2022-01-14 12:54:35,413.413 INFO:teuthology.orchestra.console:Power off smithi046 2022-01-14T12:54:35.426 INFO:teuthology.kill:2022-01-14 12:54:35,426.426 INFO:teuthology.orchestra.console:Power off smithi132 2022-01-14T12:54:43.790 INFO:teuthology.kill:2022-01-14 12:54:43,789.789 INFO:teuthology.orchestra.console:Power off for smithi046 completed 2022-01-14T12:54:43.794 INFO:teuthology.kill:2022-01-14 12:54:43,794.794 INFO:teuthology.orchestra.console:Power off for smithi132 completed 2022-01-14T12:54:43.954 INFO:teuthology.kill:2022-01-14 12:54:43,953.953 INFO:teuthology.lock.ops:unlocked: smithi132.front.sepia.ceph.com 2022-01-14T12:54:43.958 INFO:teuthology.kill:2022-01-14 12:54:43,957.957 INFO:teuthology.lock.ops:unlocked: smithi046.front.sepia.ceph.com 2022-01-14T12:56:44.372 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-01-14T12:56:44.477 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi046 but it was locked by another job: /home/teuthworker/archive/gabrioux-2022-01-14_12:33:20-rados:cephadm-wip-guits-testing-2022-01-14-001-pacific-distro-default-smithi/6616230 2022-01-14T12:56:44.478 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi132 but it was locked by another job: /home/teuthworker/archive/gabrioux-2022-01-14_12:33:20-rados:cephadm-wip-guits-testing-2022-01-14-001-pacific-distro-default-smithi/6616230