2021-10-29T04:31:50.669 INFO:root:teuthology version: 1.1.0-c56135d1
2021-10-29T04:31:50.791 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi025.front.sepia.ceph.com' reimaging
2021-10-29T04:31:50.792 INFO:teuthology.lock.ops:Updating [ubuntu@smithi025.front.sepia.ceph.com]: reset os type and version on server
2021-10-29T04:31:50.792 INFO:teuthology.lock.ops:Updating smithi025.front.sepia.ceph.com on lock server
2021-10-29T04:31:50.811 INFO:teuthology.lock.ops:Node 'ubuntu@smithi025.front.sepia.ceph.com' reimaging is complete
2021-10-29T04:31:50.812 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi138.front.sepia.ceph.com' reimaging
2021-10-29T04:31:50.812 INFO:teuthology.lock.ops:Updating [ubuntu@smithi138.front.sepia.ceph.com]: reset os type and version on server
2021-10-29T04:31:50.813 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server
2021-10-29T04:31:50.834 INFO:teuthology.lock.ops:Node 'ubuntu@smithi138.front.sepia.ceph.com' reimaging is complete
2021-10-29T04:31:51.023 INFO:teuthology.provision.fog.smithi025:Scheduling deploy of centos 8.2
2021-10-29T04:31:51.044 INFO:teuthology.provision.fog.smithi138:Scheduling deploy of centos 8.2
2021-10-29T04:31:51.365 INFO:teuthology.orchestra.console:Power off smithi025
2021-10-29T04:31:51.422 INFO:teuthology.orchestra.console:Power off smithi138
2021-10-29T04:31:51.512 INFO:teuthology.orchestra.console:Power off for smithi025 completed
2021-10-29T04:31:51.613 INFO:teuthology.orchestra.console:Power on smithi025
2021-10-29T04:32:00.075 INFO:teuthology.orchestra.console:Power on for smithi025 completed
2021-10-29T04:32:00.177 INFO:teuthology.provision.fog.smithi025:Waiting for deploy to finish
2021-10-29T04:32:03.901 INFO:teuthology.orchestra.console:Power off for smithi138 completed
2021-10-29T04:32:04.002 INFO:teuthology.orchestra.console:Power on smithi138
2021-10-29T04:32:12.368 INFO:teuthology.orchestra.console:Power on for smithi138 completed
2021-10-29T04:32:12.469 INFO:teuthology.provision.fog.smithi138:Waiting for deploy to finish
2021-10-29T04:34:46.211 ERROR:teuthology.orchestra.connection:Error authenticating with smithi025.front.sepia.ceph.com: Authentication failed.
2021-10-29T04:37:01.883 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-10-29T04:37:02.186 INFO:teuthology.provision.fog.smithi138:Node is ready
2021-10-29T04:37:02.241 INFO:teuthology.orchestra.run.smithi138.stdout:smithi138.front.sepia.ceph.com
2021-10-29T04:37:02.296 INFO:teuthology.orchestra.run.smithi138.stdout:172.21.15.138 smithi138.front.sepia.ceph.com smithi138
2021-10-29T04:37:02.532 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-10-29T04:37:02.613 INFO:teuthology.provision.fog.smithi138:Deploy complete!
2021-10-29T04:37:02.825 INFO:teuthology.provision.fog.smithi025:Node is ready
2021-10-29T04:37:02.840 INFO:teuthology.orchestra.run.smithi025.stdout:smithi025.front.sepia.ceph.com
2021-10-29T04:37:02.895 INFO:teuthology.orchestra.run.smithi025.stdout:172.21.15.25 smithi025.front.sepia.ceph.com smithi025
2021-10-29T04:37:03.215 INFO:teuthology.provision.fog.smithi025:Deploy complete!
2021-10-29T04:37:03.364 INFO:teuthology.lock.ops:Checking smithi025.front.sepia.ceph.com
2021-10-29T04:37:03.365 INFO:teuthology.lock.ops:New key found. Updating...
2021-10-29T04:37:03.383 INFO:teuthology.lock.ops:Checking smithi138.front.sepia.ceph.com
2021-10-29T04:37:03.384 INFO:teuthology.lock.ops:New key found. Updating...
2021-10-29T04:37:03.402 INFO:teuthology.lock.ops:Updating [smithi025.front.sepia.ceph.com]: set os type and version on server
2021-10-29T04:37:03.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi025.front.sepia.ceph.com'
2021-10-29T04:37:03.702 INFO:teuthology.orchestra.run.smithi025.stdout:x86_64
2021-10-29T04:37:03.757 INFO:teuthology.orchestra.run.smithi025.stdout:NAME="CentOS Linux"
2021-10-29T04:37:03.757 INFO:teuthology.orchestra.run.smithi025.stdout:VERSION="8 (Core)"
2021-10-29T04:37:03.757 INFO:teuthology.orchestra.run.smithi025.stdout:ID="centos"
2021-10-29T04:37:03.757 INFO:teuthology.orchestra.run.smithi025.stdout:ID_LIKE="rhel fedora"
2021-10-29T04:37:03.758 INFO:teuthology.orchestra.run.smithi025.stdout:VERSION_ID="8"
2021-10-29T04:37:03.758 INFO:teuthology.orchestra.run.smithi025.stdout:PLATFORM_ID="platform:el8"
2021-10-29T04:37:03.758 INFO:teuthology.orchestra.run.smithi025.stdout:PRETTY_NAME="CentOS Linux 8 (Core)"
2021-10-29T04:37:03.758 INFO:teuthology.orchestra.run.smithi025.stdout:ANSI_COLOR="0;31"
2021-10-29T04:37:03.759 INFO:teuthology.orchestra.run.smithi025.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-10-29T04:37:03.759 INFO:teuthology.orchestra.run.smithi025.stdout:HOME_URL="https://www.centos.org/"
2021-10-29T04:37:03.759 INFO:teuthology.orchestra.run.smithi025.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-10-29T04:37:03.759 INFO:teuthology.orchestra.run.smithi025.stdout:
2021-10-29T04:37:03.760 INFO:teuthology.orchestra.run.smithi025.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-10-29T04:37:03.760 INFO:teuthology.orchestra.run.smithi025.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-10-29T04:37:03.760 INFO:teuthology.orchestra.run.smithi025.stdout:REDHAT_SUPPORT_PRODUCT="centos"
2021-10-29T04:37:03.760 INFO:teuthology.orchestra.run.smithi025.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8"
2021-10-29T04:37:03.760 INFO:teuthology.orchestra.run.smithi025.stdout:
2021-10-29T04:37:03.762 INFO:teuthology.lock.ops:Updating smithi025.front.sepia.ceph.com on lock server
2021-10-29T04:37:03.780 INFO:teuthology.lock.ops:Updating [smithi138.front.sepia.ceph.com]: set os type and version on server
2021-10-29T04:37:03.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com'
2021-10-29T04:37:04.044 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64
2021-10-29T04:37:04.098 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="CentOS Linux"
2021-10-29T04:37:04.099 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="8 (Core)"
2021-10-29T04:37:04.099 INFO:teuthology.orchestra.run.smithi138.stdout:ID="centos"
2021-10-29T04:37:04.099 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="rhel fedora"
2021-10-29T04:37:04.099 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="8"
2021-10-29T04:37:04.100 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el8"
2021-10-29T04:37:04.100 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="CentOS Linux 8 (Core)"
2021-10-29T04:37:04.100 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31"
2021-10-29T04:37:04.100 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-10-29T04:37:04.100 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://www.centos.org/"
2021-10-29T04:37:04.101 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-10-29T04:37:04.101 INFO:teuthology.orchestra.run.smithi138.stdout:
2021-10-29T04:37:04.101 INFO:teuthology.orchestra.run.smithi138.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-10-29T04:37:04.101 INFO:teuthology.orchestra.run.smithi138.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-10-29T04:37:04.101 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="centos"
2021-10-29T04:37:04.102 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8"
2021-10-29T04:37:04.102 INFO:teuthology.orchestra.run.smithi138.stdout:
2021-10-29T04:37:04.102 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server
2021-10-29T04:37:04.198 INFO:teuthology.dispatcher.supervisor:Running job 6465862
2021-10-29T04:37:04.198 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology -v --owner scheduled_nojha@teuthology --archive /home/teuthworker/archive/nojha-2021-10-28_22:48:27-rados-wip-10-28-2021-distro-basic-smithi/6465862 --name nojha-2021-10-28_22:48:27-rados-wip-10-28-2021-distro-basic-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-centos_8.2_container_tools_3.0 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/nojha-2021-10-28_22:48:27-rados-wip-10-28-2021-distro-basic-smithi/6465862/orig.config.yaml
2021-10-29T04:37:04.202 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nojha-2021-10-28_22:48:27-rados-wip-10-28-2021-distro-basic-smithi/6465862
2021-10-29T04:37:04.203 INFO:teuthology.dispatcher.supervisor:Job PID: 16440
2021-10-29T04:37:04.203 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-10-29T16:37:24.466 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-10-29T16:37:24.662 INFO:teuthology.kill:Killing Pids: {16440}
2021-10-29T16:37:25.825 INFO:teuthology.task.internal:roles: ubuntu@smithi025.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2021-10-29T16:37:25.825 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2021-10-29T16:37:25.826 INFO:teuthology.misc:Compressing logs...
2021-10-29T16:37:25.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi025.front.sepia.ceph.com'
2021-10-29T16:37:26.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com'
2021-10-29T16:37:26.066 INFO:teuthology.orchestra.run.smithi025.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2021-10-29T16:37:26.324 INFO:teuthology.orchestra.run.smithi138.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2021-10-29T16:37:26.364 INFO:teuthology.orchestra.run.smithi025.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-10-29T16:37:26.365 INFO:teuthology.orchestra.run.smithi025.stderr:tar: Error is not recoverable: exiting now
2021-10-29T16:37:26.421 INFO:teuthology.orchestra.run.smithi138.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-10-29T16:37:26.422 INFO:teuthology.orchestra.run.smithi138.stderr:tar: Error is not recoverable: exiting now
2021-10-29T16:37:26.423 INFO:teuthology.misc:Compressing logs...
2021-10-29T16:37:27.016 INFO:teuthology.orchestra.run.smithi138.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-10-29T16:37:27.133 INFO:teuthology.orchestra.run.smithi025.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-10-29T16:37:27.520 INFO:teuthology.misc:Compressing logs...
2021-10-29T16:37:45.865 INFO:teuthology.orchestra.run.smithi025.stderr:gzip: /var/log/ceph/311d705c-3873-11ec-8c28-001a4aab830c/ceph-mgr.smithi025.vmevqu.log: file size changed while zipping
2021-10-29T16:37:55.609 INFO:teuthology.orchestra.run.smithi138.stderr:gzip: /var/log/ceph/311d705c-3873-11ec-8c28-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2021-10-29T16:38:02.615 INFO:teuthology.orchestra.run.smithi138.stderr:gzip: /var/log/ceph/311d705c-3873-11ec-8c28-001a4aab830c/ceph-mon.smithi138.log: file size changed while zipping
2021-10-29T16:38:07.177 INFO:teuthology.orchestra.run.smithi025.stderr:gzip: /var/log/ceph/311d705c-3873-11ec-8c28-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2021-10-29T16:38:30.316 INFO:teuthology.orchestra.run.smithi138.stderr:gzip: /var/log/ceph/311d705c-3873-11ec-8c28-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2021-10-29T16:38:37.958 INFO:teuthology.orchestra.run.smithi025.stderr:gzip: /var/log/ceph/311d705c-3873-11ec-8c28-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2021-10-29T16:38:54.029 INFO:teuthology.orchestra.run.smithi138.stderr:gzip: /var/log/ceph/311d705c-3873-11ec-8c28-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2021-10-29T16:39:16.310 INFO:teuthology.orchestra.run.smithi025.stderr:gzip: /var/log/ceph/311d705c-3873-11ec-8c28-001a4aab830c/ceph-mon.smithi025.log: file size changed while zipping
2021-10-29T16:39:25.143 INFO:teuthology.orchestra.run.smithi138.stderr:gzip: /var/log/ceph/311d705c-3873-11ec-8c28-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2021-10-29T16:39:47.003 INFO:teuthology.orchestra.run.smithi025.stderr:gzip: /var/log/ceph/311d705c-3873-11ec-8c28-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2021-10-29T16:40:16.560 INFO:teuthology.orchestra.run.smithi025.stderr:tar: .: file changed as we read it
2021-10-29T16:40:40.155 INFO:teuthology.kill:No teuthology processes running
2021-10-29T16:40:41.351 INFO:teuthology.kill:Nuking machines: ['smithi025', 'smithi138']
2021-10-29T16:40:42.404 INFO:teuthology.kill:2021-10-29 16:40:42,403.403 INFO:teuthology.nuke:targets:
2021-10-29T16:40:42.405 INFO:teuthology.kill:  smithi025.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCv/jM1qgYStEcf8yxRRE57a4/5OUWFFhrIQNcMWMEHp+pZk/kX/9l0S/hF+XMXFboKWikHSX62i3uh9lprsYdTg2RMtg43UDSnq3fAwHOPzhiDPZ0kJBbCETG6JxetJOpZan/r8SM38g2FSTlC4Q7YwgyRM5ZIsYs9xwNr0Yqfs4OIKqa+5ghffQvBi6eSeuHy8H17mvRO1BPyeAdGlUqyf0SSTvyItO6eUG6ry8GVwXAOWROR4fJiPcy0ta5N611ZVeczV0HFOfkylVe7zelofiPz/M21itoV+naEkKBTzuRnoWPCOlDo8mnq8pbbmHnfX7U2ubA02MwlUqoCPgHQska9hxZ9EuqfvGxDBQ8kAYP+3fn7Z0ND3lpLtKdH9SKNo31UU4IpXGgWFsE8WqMIfpZJwRSJyTIi7KFbizRPLexDM37mr0MC8F1C40E7yOOTJJIva1CDX7fyOJAHJdi1FAZj32Wcifxa5ZQpt06Zxrip5LCh6DGdfMKpwhheQH0=
2021-10-29T16:40:42.406 INFO:teuthology.kill:  smithi138.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUKVRZ2TZPEY0nNlhLKxJQG1+fztDNgL0/qpcUqWKe2eIx71V7tyfU7m9nPMo6QqIELfcsxEKSm1XxRhSNorbZ7lvqvIeHMZYyx2CQPNkkVVRHU5WkNAFq3G54dQDo9szCvjES3u5Lze0FeOMFysOSZfWgyeHw6VKJJQPwDfptc3va4xezVEFwTTaRZRinj6Q3+/7NLguwyx8ncZ4qvdWG0qkvh1qNnvLHWa7xHhMtsUpU9x6Iw9h9TcymQLNmtsR+n2n3SIBLt7mS+f+liH0/r/dKk7cYKCiiJaU75FfMgW20EZRmAIFdwlAnAB7rQJ+B5+/sSsrKWhDf7j5S4bAhxW/gr4OULs3MZtBQ8soQqm/bHTaRqEArqanHOmd0paAQ7V0uIMz0xiGks0DS+yRfQJ0s11pZKe5cyaOYGX0TeYtdVfvoH6e7rP7sZbXcggNNKIwgaqSHJroNu+25B1OfHcv2lr4qLFKbre5fgEhJWcIDfxnKN3DrQLAQZVMUKTU=
2021-10-29T16:40:42.435 INFO:teuthology.kill:2021-10-29 16:40:42,435.435 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-10-29T16:40:42.441 INFO:teuthology.kill:2021-10-29 16:40:42,440.440 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-10-29T16:40:42.520 INFO:teuthology.kill:2021-10-29 16:40:42,520.520 INFO:teuthology.orchestra.console:Power off smithi138
2021-10-29T16:40:42.531 INFO:teuthology.kill:2021-10-29 16:40:42,531.531 INFO:teuthology.orchestra.console:Power off smithi025
2021-10-29T16:40:55.016 INFO:teuthology.kill:2021-10-29 16:40:55,015.015 INFO:teuthology.orchestra.console:Power off for smithi138 completed
2021-10-29T16:40:55.021 INFO:teuthology.kill:2021-10-29 16:40:55,021.021 INFO:teuthology.orchestra.console:Power off for smithi025 completed
2021-10-29T16:40:55.172 INFO:teuthology.kill:2021-10-29 16:40:55,171.171 INFO:teuthology.lock.ops:unlocked: smithi138.front.sepia.ceph.com
2021-10-29T16:40:55.174 INFO:teuthology.kill:2021-10-29 16:40:55,174.174 INFO:teuthology.lock.ops:unlocked: smithi025.front.sepia.ceph.com
2021-10-29T16:42:55.441 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-10-29T16:42:55.541 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi025 but it was locked by another job: /home/teuthworker/archive/matan-2021-10-29_13:13:12-rgw-wip-matanb-lua-rgw-map-2021-10-27A-distro-basic-smithi/6467001
2021-10-29T16:42:55.542 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi138 but it was locked by another job: /home/teuthworker/archive/matan-2021-10-29_13:13:12-rgw-wip-matanb-lua-rgw-map-2021-10-27A-distro-basic-smithi/6467001