2022-01-24T22:47:31.974 INFO:root:teuthology version: 1.1.0-7c0cb867
2022-01-24T22:47:32.103 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi104.front.sepia.ceph.com' reimaging
2022-01-24T22:47:32.104 INFO:teuthology.lock.ops:Updating [ubuntu@smithi104.front.sepia.ceph.com]: reset os type and version on server
2022-01-24T22:47:32.104 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server
2022-01-24T22:47:32.123 INFO:teuthology.lock.ops:Node 'ubuntu@smithi104.front.sepia.ceph.com' reimaging is complete
2022-01-24T22:47:32.123 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi190.front.sepia.ceph.com' reimaging
2022-01-24T22:47:32.123 INFO:teuthology.lock.ops:Updating [ubuntu@smithi190.front.sepia.ceph.com]: reset os type and version on server
2022-01-24T22:47:32.124 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server
2022-01-24T22:47:32.146 INFO:teuthology.lock.ops:Node 'ubuntu@smithi190.front.sepia.ceph.com' reimaging is complete
2022-01-24T22:47:32.333 INFO:teuthology.provision.fog.smithi104:Scheduling deploy of centos 8.stream
2022-01-24T22:47:32.348 INFO:teuthology.provision.fog.smithi190:Scheduling deploy of centos 8.stream
2022-01-24T22:47:32.737 INFO:teuthology.orchestra.console:Power off smithi104
2022-01-24T22:47:32.751 INFO:teuthology.orchestra.console:Power off smithi190
2022-01-24T22:47:41.114 INFO:teuthology.orchestra.console:Power off for smithi190 completed
2022-01-24T22:47:41.215 INFO:teuthology.orchestra.console:Power on smithi190
2022-01-24T22:47:45.214 INFO:teuthology.orchestra.console:Power off for smithi104 completed
2022-01-24T22:47:45.315 INFO:teuthology.orchestra.console:Power on smithi104
2022-01-24T22:47:49.568 INFO:teuthology.orchestra.console:Power on for smithi104 completed
2022-01-24T22:47:49.574 INFO:teuthology.orchestra.console:Power on for smithi190 completed
2022-01-24T22:47:49.670 INFO:teuthology.provision.fog.smithi104:Waiting for deploy to finish
2022-01-24T22:47:49.674 INFO:teuthology.provision.fog.smithi190:Waiting for deploy to finish
2022-01-24T22:51:56.332 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-01-24T22:51:56.645 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-01-24T22:52:21.373 INFO:teuthology.provision.fog.smithi104:Node is ready
2022-01-24T22:52:21.394 INFO:teuthology.orchestra.run.smithi104.stdout:smithi104.front.sepia.ceph.com
2022-01-24T22:52:21.451 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104 smithi104.front.sepia.ceph.com smithi104
2022-01-24T22:52:21.808 INFO:teuthology.provision.fog.smithi104:Deploy complete!
2022-01-24T22:52:22.371 INFO:teuthology.provision.fog.smithi190:Node is ready
2022-01-24T22:52:22.391 INFO:teuthology.orchestra.run.smithi190.stdout:smithi190.front.sepia.ceph.com
2022-01-24T22:52:22.452 INFO:teuthology.orchestra.run.smithi190.stdout:172.21.15.190 smithi190.front.sepia.ceph.com smithi190
2022-01-24T22:52:22.807 INFO:teuthology.provision.fog.smithi190:Deploy complete!
2022-01-24T22:52:22.957 INFO:teuthology.lock.ops:Checking smithi104.front.sepia.ceph.com
2022-01-24T22:52:22.958 INFO:teuthology.lock.ops:New key found. Updating...
2022-01-24T22:52:22.990 INFO:teuthology.lock.ops:Checking smithi190.front.sepia.ceph.com
2022-01-24T22:52:22.990 INFO:teuthology.lock.ops:New key found. Updating...
2022-01-24T22:52:23.010 INFO:teuthology.lock.ops:Updating [smithi104.front.sepia.ceph.com]: set os type and version on server
2022-01-24T22:52:23.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com'
2022-01-24T22:52:23.212 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64
2022-01-24T22:52:23.270 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream"
2022-01-24T22:52:23.270 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="8"
2022-01-24T22:52:23.271 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos"
2022-01-24T22:52:23.271 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora"
2022-01-24T22:52:23.271 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="8"
2022-01-24T22:52:23.271 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el8"
2022-01-24T22:52:23.272 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 8"
2022-01-24T22:52:23.272 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31"
2022-01-24T22:52:23.272 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-01-24T22:52:23.272 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/"
2022-01-24T22:52:23.273 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-01-24T22:52:23.273 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-01-24T22:52:23.273 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-01-24T22:52:23.274 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server
2022-01-24T22:52:23.293 INFO:teuthology.lock.ops:Updating [smithi190.front.sepia.ceph.com]: set os type and version on server
2022-01-24T22:52:23.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com'
2022-01-24T22:52:23.487 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64
2022-01-24T22:52:23.543 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream"
2022-01-24T22:52:23.543 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="8"
2022-01-24T22:52:23.544 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos"
2022-01-24T22:52:23.544 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora"
2022-01-24T22:52:23.544 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="8"
2022-01-24T22:52:23.544 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el8"
2022-01-24T22:52:23.545 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 8"
2022-01-24T22:52:23.545 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31"
2022-01-24T22:52:23.545 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-01-24T22:52:23.545 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/"
2022-01-24T22:52:23.546 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-01-24T22:52:23.546 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-01-24T22:52:23.546 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-01-24T22:52:23.546 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server
2022-01-24T22:52:23.640 INFO:teuthology.dispatcher.supervisor:Running job 6638251
2022-01-24T22:52:23.640 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-01-24_17:43:02-rados-wip-yuri2-testing-2022-01-21-0949-pacific-distro-default-smithi/6638251 --name yuriw-2022-01-24_17:43:02-rados-wip-yuri2-testing-2022-01-21-0949-pacific-distro-default-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2022-01-24_17:43:02-rados-wip-yuri2-testing-2022-01-21-0949-pacific-distro-default-smithi/6638251/orig.config.yaml
2022-01-24T22:52:23.644 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-01-24_17:43:02-rados-wip-yuri2-testing-2022-01-21-0949-pacific-distro-default-smithi/6638251
2022-01-24T22:52:23.645 INFO:teuthology.dispatcher.supervisor:Job PID: 18031
2022-01-24T22:52:23.645 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-01-25T05:22:53.544 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-01-25T05:22:53.732 INFO:teuthology.kill:Killing Pids: {18031}
2022-01-25T05:22:54.970 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-01-25T05:22:54.971 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-01-25T05:22:54.972 INFO:teuthology.misc:Compressing logs...
2022-01-25T05:22:54.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com'
2022-01-25T05:22:55.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com'
2022-01-25T05:22:55.213 INFO:teuthology.orchestra.run.smithi104.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-01-25T05:22:55.405 INFO:teuthology.orchestra.run.smithi190.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-01-25T05:22:55.445 INFO:teuthology.orchestra.run.smithi104.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-01-25T05:22:55.445 INFO:teuthology.orchestra.run.smithi104.stderr:tar: Error is not recoverable: exiting now
2022-01-25T05:22:55.561 INFO:teuthology.orchestra.run.smithi190.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-01-25T05:22:55.562 INFO:teuthology.orchestra.run.smithi190.stderr:tar: Error is not recoverable: exiting now
2022-01-25T05:22:55.563 INFO:teuthology.misc:Compressing logs...
2022-01-25T05:22:55.863 INFO:teuthology.misc:Compressing logs...
2022-01-25T05:22:58.514 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/6e87afbc-7d69-11ec-8c35-001a4aab830c/ceph-mon.smithi104.log: file size changed while zipping
2022-01-25T05:22:59.043 INFO:teuthology.orchestra.run.smithi190.stderr:gzip: /var/log/ceph/6e87afbc-7d69-11ec-8c35-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-01-25T05:22:59.582 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/6e87afbc-7d69-11ec-8c35-001a4aab830c/ceph-mgr.smithi104.jxjwqn.log: file size changed while zipping
2022-01-25T05:23:01.930 INFO:teuthology.orchestra.run.smithi190.stderr:gzip: /var/log/ceph/6e87afbc-7d69-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-01-25T05:23:02.372 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/6e87afbc-7d69-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-01-25T05:23:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/6e87afbc-7d69-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-01-25T05:23:04.988 INFO:teuthology.orchestra.run.smithi190.stderr:gzip: /var/log/ceph/6e87afbc-7d69-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-01-25T05:23:06.765 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/6e87afbc-7d69-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-01-25T05:23:07.591 INFO:teuthology.orchestra.run.smithi190.stderr:gzip: /var/log/ceph/6e87afbc-7d69-11ec-8c35-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-01-25T05:23:08.577 INFO:teuthology.orchestra.run.smithi190.stderr:gzip: /var/log/ceph/6e87afbc-7d69-11ec-8c35-001a4aab830c/ceph-mon.smithi190.log: file size changed while zipping
2022-01-25T05:23:09.526 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/6e87afbc-7d69-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-01-25T05:23:26.551 INFO:teuthology.orchestra.run.smithi104.stderr:tar: .: file changed as we read it
2022-01-25T05:23:40.656 INFO:teuthology.kill:No teuthology processes running
2022-01-25T05:23:41.813 INFO:teuthology.kill:Nuking machines: ['smithi104', 'smithi190']
2022-01-25T05:23:42.791 INFO:teuthology.kill:2022-01-25 05:23:42,791.791 INFO:teuthology.nuke:targets:
2022-01-25T05:23:42.792 INFO:teuthology.kill:  smithi104.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwM33wCunDVYB5kIJ4Mgqdewm7EjaQM7R1aDp+6TPVJwZcPz8wojzVdIFbe+b00kPh9pNfvrn+h7fGs6hfZ9u5LfUWr44iUp6GF4nB9U9qD7xt15bLnmZAmJZ7wR17ds+78zpUhi3vAy7BUMbxZz44f8G4K4PtaFcci2M4YVU8hqm4VOhMK6zeuttssKxOPt6qqm6hCe+j2wBRlNGixRTqM0pjHjp3oYbZ+2fehXnotV+AXIW71faNFNhFPVo7XkQiIlzzGmDt3mrrzr9g+ZNZr8wKle7QLBjL9zsrt7OMf26K+mwdc6FXNv5AXfJzs8xsavIUnYoDLu5iDUfo/BfCKIpgdqhif8WgYusPtNVoms5/Jsvm/VsPTpJJnqVI/AuV+ECRNyNLMZ0T6shK3AyiO27tHnaymFkE/UZjnht8Cq+/Ht6/T0P6iNKCVDeNJhlFoY3m9ztyStljrLb+NscVOsRno9iY7C7U07z0djIqUFJUilDj0QMjkjrOyqY9hIs=
2022-01-25T05:23:42.793 INFO:teuthology.kill:  smithi190.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9iDp2X/khXBVRsLUgc7a9II6aN3WlHEvDrtTxafJmMRmPXky7qchPql32xxU9Oc0/oz/uUvurJpn9B7VA3bchUw9gaAMalItzKDjwagxbqHSeY3WNbHH0aahQ5tRiU9FnJgNHnDELwUBt85ZqBf3crk2Bngqg7OYwHLwCwUr7OBlc2vOoqa5ObfwYQu0m+hh1GvNa/dABm0kWKn4Aes3F8bBYVFTI0McIxOY6c2usVPp8ZYsUqjH7z7AlTkHhXrzfgw4i7dfCv1g7VanMaRDPAALirxx9+Vt9dX4pSB5q2w33Z31rFOGEdGh3ZytyE6MrqinSHYGmuviK2IgQ3pw4dbQVG/QpCTqXH4qOo5KQD3VKw0mfRsd2jrglRzzfu4DBzvBaWnAzYks5TwnAijtpntBlhoRQKGbTOoWP5DYy0wKcTZ7FpL9sjM14E4jKnjegDHJKLcqj1EzdDN9EF8aaK9hCc3Kht3XzwyhHlr4I1O+pTGUNqoQXtccIrtpvblU=
2022-01-25T05:23:42.819 INFO:teuthology.kill:2022-01-25 05:23:42,819.819 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-01-25T05:23:42.830 INFO:teuthology.kill:2022-01-25 05:23:42,830.830 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-01-25T05:23:42.890 INFO:teuthology.kill:2022-01-25 05:23:42,890.890 INFO:teuthology.orchestra.console:Power off smithi104
2022-01-25T05:23:42.907 INFO:teuthology.kill:2022-01-25 05:23:42,906.906 INFO:teuthology.orchestra.console:Power off smithi190
2022-01-25T05:23:51.261 INFO:teuthology.kill:2022-01-25 05:23:51,261.261 INFO:teuthology.orchestra.console:Power off for smithi104 completed
2022-01-25T05:23:51.406 INFO:teuthology.kill:2022-01-25 05:23:51,405.405 INFO:teuthology.lock.ops:unlocked: smithi104.front.sepia.ceph.com
2022-01-25T05:23:55.397 INFO:teuthology.kill:2022-01-25 05:23:55,397.397 INFO:teuthology.orchestra.console:Power off for smithi190 completed
2022-01-25T05:23:55.551 INFO:teuthology.kill:2022-01-25 05:23:55,551.551 INFO:teuthology.lock.ops:unlocked: smithi190.front.sepia.ceph.com
2022-01-25T05:25:55.845 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-01-25T05:25:55.944 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi104 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-01-25_03:40:28-rados-wip-yuri4-testing-2022-01-24-1706-pacific-distro-default-smithi/6639189
2022-01-25T05:25:55.944 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi190 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-01-25_03:40:28-rados-wip-yuri4-testing-2022-01-24-1706-pacific-distro-default-smithi/6639189