2021-12-10T20:29:16.108 INFO:root:teuthology version: 1.1.0-95a7d479 2021-12-10T20:29:16.233 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi118.front.sepia.ceph.com' reimaging 2021-12-10T20:29:16.234 INFO:teuthology.lock.ops:Updating [ubuntu@smithi118.front.sepia.ceph.com]: reset os type and version on server 2021-12-10T20:29:16.235 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2021-12-10T20:29:16.254 INFO:teuthology.lock.ops:Node 'ubuntu@smithi118.front.sepia.ceph.com' reimaging is complete 2021-12-10T20:29:16.254 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging 2021-12-10T20:29:16.254 INFO:teuthology.lock.ops:Updating [ubuntu@smithi142.front.sepia.ceph.com]: reset os type and version on server 2021-12-10T20:29:16.255 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2021-12-10T20:29:16.275 INFO:teuthology.lock.ops:Node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging is complete 2021-12-10T20:29:16.466 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of centos 8.stream 2021-12-10T20:29:16.470 INFO:teuthology.provision.fog.smithi118:Scheduling deploy of centos 8.stream 2021-12-10T20:29:16.795 INFO:teuthology.orchestra.console:Power off smithi142 2021-12-10T20:29:16.811 INFO:teuthology.orchestra.console:Power off smithi118 2021-12-10T20:29:25.167 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2021-12-10T20:29:25.173 INFO:teuthology.orchestra.console:Power off for smithi118 completed 2021-12-10T20:29:25.269 INFO:teuthology.orchestra.console:Power on smithi142 2021-12-10T20:29:25.279 INFO:teuthology.orchestra.console:Power on smithi118 2021-12-10T20:29:33.641 INFO:teuthology.orchestra.console:Power on for smithi142 completed 2021-12-10T20:29:33.678 INFO:teuthology.orchestra.console:Power on for smithi118 completed 2021-12-10T20:29:33.779 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish 2021-12-10T20:29:33.781 INFO:teuthology.provision.fog.smithi118:Waiting for deploy to finish 2021-12-10T20:34:24.187 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-10T20:34:24.476 INFO:teuthology.provision.fog.smithi142:Node is ready 2021-12-10T20:34:24.494 INFO:teuthology.orchestra.run.smithi142.stdout:smithi142.front.sepia.ceph.com 2021-12-10T20:34:24.552 INFO:teuthology.orchestra.run.smithi142.stdout:172.21.15.142 smithi142.front.sepia.ceph.com smithi142 2021-12-10T20:34:24.878 INFO:teuthology.provision.fog.smithi142:Deploy complete! 2021-12-10T20:34:51.597 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-10T20:34:51.913 INFO:teuthology.provision.fog.smithi118:Node is ready 2021-12-10T20:34:51.930 INFO:teuthology.orchestra.run.smithi118.stdout:smithi118.front.sepia.ceph.com 2021-12-10T20:34:51.988 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118 smithi118.front.sepia.ceph.com smithi118 2021-12-10T20:34:52.335 INFO:teuthology.provision.fog.smithi118:Deploy complete! 2021-12-10T20:34:52.492 INFO:teuthology.lock.ops:Checking smithi118.front.sepia.ceph.com 2021-12-10T20:34:52.493 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-10T20:34:52.512 INFO:teuthology.lock.ops:Checking smithi142.front.sepia.ceph.com 2021-12-10T20:34:52.512 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-10T20:34:52.530 INFO:teuthology.lock.ops:Updating [smithi118.front.sepia.ceph.com]: set os type and version on server 2021-12-10T20:34:52.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2021-12-10T20:34:52.717 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2021-12-10T20:34:52.772 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Stream" 2021-12-10T20:34:52.773 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="8" 2021-12-10T20:34:52.773 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos" 2021-12-10T20:34:52.773 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora" 2021-12-10T20:34:52.774 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="8" 2021-12-10T20:34:52.774 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el8" 2021-12-10T20:34:52.774 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Stream 8" 2021-12-10T20:34:52.774 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2021-12-10T20:34:52.775 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-12-10T20:34:52.775 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://centos.org/" 2021-12-10T20:34:52.775 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-10T20:34:52.775 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-10T20:34:52.776 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-12-10T20:34:52.777 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2021-12-10T20:34:52.796 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server 2021-12-10T20:34:52.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2021-12-10T20:34:53.044 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2021-12-10T20:34:53.101 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="CentOS Stream" 2021-12-10T20:34:53.101 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="8" 2021-12-10T20:34:53.102 INFO:teuthology.orchestra.run.smithi142.stdout:ID="centos" 2021-12-10T20:34:53.102 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="rhel fedora" 2021-12-10T20:34:53.102 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="8" 2021-12-10T20:34:53.103 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el8" 2021-12-10T20:34:53.103 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="CentOS Stream 8" 2021-12-10T20:34:53.103 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2021-12-10T20:34:53.103 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-12-10T20:34:53.104 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://centos.org/" 2021-12-10T20:34:53.104 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-10T20:34:53.104 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-10T20:34:53.104 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-12-10T20:34:53.105 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2021-12-10T20:34:53.203 INFO:teuthology.dispatcher.supervisor:Running job 6557505 2021-12-10T20:34:53.204 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-12-10_16:36:47-rados-wip-yuri2-testing-2021-12-09-1306-distro-default-smithi/6557505 --name yuriw-2021-12-10_16:36:47-rados-wip-yuri2-testing-2021-12-09-1306-distro-default-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2021-12-10_16:36:47-rados-wip-yuri2-testing-2021-12-09-1306-distro-default-smithi/6557505/orig.config.yaml 2021-12-10T20:34:53.207 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-12-10_16:36:47-rados-wip-yuri2-testing-2021-12-09-1306-distro-default-smithi/6557505 2021-12-10T20:34:53.208 INFO:teuthology.dispatcher.supervisor:Job PID: 21694 2021-12-10T20:34:53.208 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-12-11T08:35:47.825 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-12-11T08:35:48.005 INFO:teuthology.kill:Killing Pids: {21694} 2021-12-11T08:35:49.166 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-12-11T08:35:49.166 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-12-11T08:35:49.167 INFO:teuthology.misc:Compressing logs... 2021-12-11T08:35:49.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2021-12-11T08:35:49.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2021-12-11T08:35:49.410 INFO:teuthology.orchestra.run.smithi118.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-12-11T08:35:49.598 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-12-11T08:35:49.640 INFO:teuthology.orchestra.run.smithi118.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-12-11T08:35:49.641 INFO:teuthology.orchestra.run.smithi118.stderr:tar: Error is not recoverable: exiting now 2021-12-11T08:35:49.710 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-12-11T08:35:49.710 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now 2021-12-11T08:35:49.711 INFO:teuthology.misc:Compressing logs... 2021-12-11T08:35:50.079 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-12-11T08:35:50.273 INFO:teuthology.misc:Compressing logs... 2021-12-11T08:35:54.461 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/2e5cff7a-59f9-11ec-8c31-001a4aab830c/ceph-mon.smithi142.log: file size changed while zipping 2021-12-11T08:35:58.057 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/2e5cff7a-59f9-11ec-8c31-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2021-12-11T08:36:02.775 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/2e5cff7a-59f9-11ec-8c31-001a4aab830c/ceph-osd.7.log: file size changed while zipping 2021-12-11T08:36:05.855 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/2e5cff7a-59f9-11ec-8c31-001a4aab830c/ceph-mon.smithi118.log: file size changed while zipping 2021-12-11T08:36:09.967 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/2e5cff7a-59f9-11ec-8c31-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2021-12-11T08:36:11.741 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/2e5cff7a-59f9-11ec-8c31-001a4aab830c/ceph-mgr.smithi142.bhtyok.log: file size changed while zipping 2021-12-11T08:36:13.121 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/2e5cff7a-59f9-11ec-8c31-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2021-12-11T08:36:18.568 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/2e5cff7a-59f9-11ec-8c31-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2021-12-11T08:36:20.021 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/2e5cff7a-59f9-11ec-8c31-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2021-12-11T08:36:26.510 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/2e5cff7a-59f9-11ec-8c31-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2021-12-11T08:36:26.808 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/2e5cff7a-59f9-11ec-8c31-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2021-12-11T08:37:14.307 INFO:teuthology.kill:No teuthology processes running 2021-12-11T08:37:15.530 INFO:teuthology.kill:Nuking machines: ['smithi118', 'smithi142'] 2021-12-11T08:37:16.613 INFO:teuthology.kill:2021-12-11 08:37:16,613.613 INFO:teuthology.nuke:targets: 2021-12-11T08:37:16.614 INFO:teuthology.kill: smithi118.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEj1Yfa2D1cLMyMTwvb0S5uwy7nc5qrtgJTFehoicHkcUeoBgoEUinajE9yAQffna1pauG3jke+ZH61rYRljtk4s7wTiDWREtR41saFSiLQ9O8vgc5kUxz2kZhSgm++tfs7S27JxNQrN0cG8FQH4kbILdi3KHv/HMk6Fq6jbt4OA1CByPDzqp475kr1bK+hSNdupBc9DfB1CiZ5THAFehZ89DHLtz0Vov4HPD7I2MwYm4eYWWc3CbnQHWFSxWOWu8Qjw3jwbaOakeePJmvDiRxTIn8huGkVjkmXueK5XgRyDTIib+MEL5rM9NZenYAm9zmB/kyg4eAANlW/kpTECmBr9y+NKmYzPvdOykYP7nL2p7m4BQZEf/yrNwfizz5+HbK1Vd4khh6tugX7UmoYebpAGt06cBt+WX9i49pyoVYt9OvSct0brZeecfeVByAwGQLctEv3Tb8G05Z3Bxb8DIeUtQi7EohJcx+g9dwdYqqLXoc20Hh2qWumEIt/yVbZh0= 2021-12-11T08:37:16.614 INFO:teuthology.kill: smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDm2qwiv0EKmuQNF3rgqInF2GQM77cdBlOz75Byha91sPKb51n4I2EuPAza5kaQCsmcfecn6O8zGtoldQJ3tvLUEgha0pnxAmm/wR95ea137Yg1b9Al+2FuQpmsc+krZzEhmS6DejHYgOd/k8cDNCYyVPbl4Is29WKZIQSwHMpvf/JsKCMoBufj54tAyBsxSaiCjkpAswG201Rtg567fZ5taNBDKn0Ep3v9783pAiErO56c5caSkesR7iUvVTAMFi8DPeXX1PLLtrMUP6WZM/0xURJtjEjdzEFHpHC2+v/6NqfzK5PXBV0H0NbCam/P2GTeXqwl3/lH8at/+uPG98Ec3rO/rGFR0eRFGpwADyLqR4Hf2yS8Qw0oqWfoj3VpRAWm/T9iE/o7RedFyaYCZClzPIDfutrrW4UtcpjM0Ypqr4XbJi4jXdeiuKv/XSkfRDykrFGPLKAjvu2so5dznEpP8on15ly8MZBahl2MT0NisOfdNISDqHYLHII3gSAXdhM= 2021-12-11T08:37:16.641 INFO:teuthology.kill:2021-12-11 08:37:16,641.641 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-11T08:37:16.645 INFO:teuthology.kill:2021-12-11 08:37:16,645.645 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-11T08:37:16.732 INFO:teuthology.kill:2021-12-11 08:37:16,732.732 INFO:teuthology.orchestra.console:Power off smithi142 2021-12-11T08:37:16.748 INFO:teuthology.kill:2021-12-11 08:37:16,747.747 INFO:teuthology.orchestra.console:Power off smithi118 2021-12-11T08:37:25.114 INFO:teuthology.kill:2021-12-11 08:37:25,114.114 INFO:teuthology.orchestra.console:Power off for smithi118 completed 2021-12-11T08:37:25.266 INFO:teuthology.kill:2021-12-11 08:37:25,264.264 INFO:teuthology.lock.ops:unlocked: smithi118.front.sepia.ceph.com 2021-12-11T08:37:29.220 INFO:teuthology.kill:2021-12-11 08:37:29,220.220 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2021-12-11T08:37:29.366 INFO:teuthology.kill:2021-12-11 08:37:29,366.366 INFO:teuthology.lock.ops:unlocked: smithi142.front.sepia.ceph.com 2021-12-11T08:39:29.678 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-12-11T08:39:29.779 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi118 but it was locked by another job: /home/teuthworker/archive/teuthology-2021-12-10_03:20:03-krbd-master-testing-default-smithi/6556115 2021-12-11T08:39:29.779 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi142 but it was locked by another job: /home/teuthworker/archive/teuthology-2021-12-10_03:20:03-krbd-master-testing-default-smithi/6556114