2022-04-05T17:24:29.799 INFO:root:teuthology version: 1.1.0-d877b023
2022-04-05T17:24:29.917 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi065.front.sepia.ceph.com' reimaging
2022-04-05T17:24:29.918 INFO:teuthology.lock.ops:Updating [ubuntu@smithi065.front.sepia.ceph.com]: reset os type and version on server
2022-04-05T17:24:29.918 INFO:teuthology.lock.ops:Updating smithi065.front.sepia.ceph.com on lock server
2022-04-05T17:24:29.938 INFO:teuthology.lock.ops:Node 'ubuntu@smithi065.front.sepia.ceph.com' reimaging is complete
2022-04-05T17:24:29.938 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi073.front.sepia.ceph.com' reimaging
2022-04-05T17:24:29.939 INFO:teuthology.lock.ops:Updating [ubuntu@smithi073.front.sepia.ceph.com]: reset os type and version on server
2022-04-05T17:24:29.939 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server
2022-04-05T17:24:29.963 INFO:teuthology.lock.ops:Node 'ubuntu@smithi073.front.sepia.ceph.com' reimaging is complete
2022-04-05T17:24:30.154 INFO:teuthology.provision.fog.smithi065:Scheduling deploy of centos 8.stream
2022-04-05T17:24:30.169 INFO:teuthology.provision.fog.smithi073:Scheduling deploy of centos 8.stream
2022-04-05T17:24:30.470 INFO:teuthology.orchestra.console:Power off smithi065
2022-04-05T17:24:30.515 INFO:teuthology.orchestra.console:Power off smithi073
2022-04-05T17:24:38.853 INFO:teuthology.orchestra.console:Power off for smithi065 completed
2022-04-05T17:24:38.880 INFO:teuthology.orchestra.console:Power off for smithi073 completed
2022-04-05T17:24:38.954 INFO:teuthology.orchestra.console:Power on smithi065
2022-04-05T17:24:38.986 INFO:teuthology.orchestra.console:Power on smithi073
2022-04-05T17:24:43.213 INFO:teuthology.orchestra.console:Power on for smithi065 completed
2022-04-05T17:24:43.314 INFO:teuthology.provision.fog.smithi065:Waiting for deploy to finish
2022-04-05T17:24:47.353 INFO:teuthology.orchestra.console:Power on for smithi073 completed
2022-04-05T17:24:47.454 INFO:teuthology.provision.fog.smithi073:Waiting for deploy to finish
2022-04-05T17:29:01.978 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-04-05T17:29:27.426 INFO:teuthology.provision.fog.smithi065:Node is ready
2022-04-05T17:29:27.446 INFO:teuthology.orchestra.run.smithi065.stdout:smithi065.front.sepia.ceph.com
2022-04-05T17:29:27.504 INFO:teuthology.orchestra.run.smithi065.stdout:172.21.15.65 smithi065.front.sepia.ceph.com smithi065
2022-04-05T17:29:27.888 INFO:teuthology.provision.fog.smithi065:Deploy complete!
2022-04-05T17:29:31.489 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-04-05T17:29:31.775 INFO:teuthology.provision.fog.smithi073:Node is ready
2022-04-05T17:29:31.794 INFO:teuthology.orchestra.run.smithi073.stdout:smithi073.front.sepia.ceph.com
2022-04-05T17:29:31.853 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73 smithi073.front.sepia.ceph.com smithi073
2022-04-05T17:29:32.196 INFO:teuthology.provision.fog.smithi073:Deploy complete!
2022-04-05T17:29:32.340 INFO:teuthology.lock.ops:Checking smithi065.front.sepia.ceph.com
2022-04-05T17:29:32.341 INFO:teuthology.lock.ops:New key found. Updating...
2022-04-05T17:29:32.360 INFO:teuthology.lock.ops:Checking smithi073.front.sepia.ceph.com
2022-04-05T17:29:32.361 INFO:teuthology.lock.ops:New key found. Updating...
2022-04-05T17:29:32.378 INFO:teuthology.lock.ops:Updating [smithi065.front.sepia.ceph.com]: set os type and version on server
2022-04-05T17:29:32.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi065.front.sepia.ceph.com'
2022-04-05T17:29:32.581 INFO:teuthology.orchestra.run.smithi065.stdout:x86_64
2022-04-05T17:29:32.640 INFO:teuthology.orchestra.run.smithi065.stdout:NAME="CentOS Stream"
2022-04-05T17:29:32.640 INFO:teuthology.orchestra.run.smithi065.stdout:VERSION="8"
2022-04-05T17:29:32.640 INFO:teuthology.orchestra.run.smithi065.stdout:ID="centos"
2022-04-05T17:29:32.640 INFO:teuthology.orchestra.run.smithi065.stdout:ID_LIKE="rhel fedora"
2022-04-05T17:29:32.641 INFO:teuthology.orchestra.run.smithi065.stdout:VERSION_ID="8"
2022-04-05T17:29:32.641 INFO:teuthology.orchestra.run.smithi065.stdout:PLATFORM_ID="platform:el8"
2022-04-05T17:29:32.641 INFO:teuthology.orchestra.run.smithi065.stdout:PRETTY_NAME="CentOS Stream 8"
2022-04-05T17:29:32.641 INFO:teuthology.orchestra.run.smithi065.stdout:ANSI_COLOR="0;31"
2022-04-05T17:29:32.642 INFO:teuthology.orchestra.run.smithi065.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-04-05T17:29:32.642 INFO:teuthology.orchestra.run.smithi065.stdout:HOME_URL="https://centos.org/"
2022-04-05T17:29:32.642 INFO:teuthology.orchestra.run.smithi065.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-04-05T17:29:32.642 INFO:teuthology.orchestra.run.smithi065.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-04-05T17:29:32.643 INFO:teuthology.orchestra.run.smithi065.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-04-05T17:29:32.644 INFO:teuthology.lock.ops:Updating smithi065.front.sepia.ceph.com on lock server
2022-04-05T17:29:32.664 INFO:teuthology.lock.ops:Updating [smithi073.front.sepia.ceph.com]: set os type and version on server
2022-04-05T17:29:32.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com'
2022-04-05T17:29:32.852 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64
2022-04-05T17:29:32.909 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="CentOS Stream"
2022-04-05T17:29:32.909 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="8"
2022-04-05T17:29:32.909 INFO:teuthology.orchestra.run.smithi073.stdout:ID="centos"
2022-04-05T17:29:32.910 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE="rhel fedora"
2022-04-05T17:29:32.910 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="8"
2022-04-05T17:29:32.910 INFO:teuthology.orchestra.run.smithi073.stdout:PLATFORM_ID="platform:el8"
2022-04-05T17:29:32.910 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="CentOS Stream 8"
2022-04-05T17:29:32.911 INFO:teuthology.orchestra.run.smithi073.stdout:ANSI_COLOR="0;31"
2022-04-05T17:29:32.911 INFO:teuthology.orchestra.run.smithi073.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-04-05T17:29:32.911 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://centos.org/"
2022-04-05T17:29:32.911 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-04-05T17:29:32.911 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-04-05T17:29:32.912 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-04-05T17:29:32.912 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server
2022-04-05T17:29:33.003 INFO:teuthology.dispatcher.supervisor:Running job 6777969
2022-04-05T17:29:33.004 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-04-05_17:16:28-orch-wip-yuri2-testing-2022-03-31-1523-pacific-distro-default-smithi/6777969 --name yuriw-2022-04-05_17:16:28-orch-wip-yuri2-testing-2022-03-31-1523-pacific-distro-default-smithi --description orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2022-04-05_17:16:28-orch-wip-yuri2-testing-2022-03-31-1523-pacific-distro-default-smithi/6777969/orig.config.yaml
2022-04-05T17:29:33.007 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-04-05_17:16:28-orch-wip-yuri2-testing-2022-03-31-1523-pacific-distro-default-smithi/6777969
2022-04-05T17:29:33.007 INFO:teuthology.dispatcher.supervisor:Job PID: 3377
2022-04-05T17:29:33.008 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-04-05T23:59:59.131 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-04-05T23:59:59.314 INFO:teuthology.kill:Killing Pids: {3377}
2022-04-06T00:00:00.407 INFO:teuthology.task.internal:roles: ubuntu@smithi065.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-04-06T00:00:00.408 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-04-06T00:00:00.408 INFO:teuthology.misc:Compressing logs...
2022-04-06T00:00:00.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi065.front.sepia.ceph.com'
2022-04-06T00:00:00.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com'
2022-04-06T00:00:00.660 INFO:teuthology.orchestra.run.smithi065.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-04-06T00:00:00.826 INFO:teuthology.orchestra.run.smithi073.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-04-06T00:00:00.879 INFO:teuthology.orchestra.run.smithi065.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-04-06T00:00:00.879 INFO:teuthology.orchestra.run.smithi065.stderr:tar: Error is not recoverable: exiting now
2022-04-06T00:00:00.956 INFO:teuthology.orchestra.run.smithi073.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-04-06T00:00:00.956 INFO:teuthology.orchestra.run.smithi073.stderr:tar: Error is not recoverable: exiting now
2022-04-06T00:00:00.958 INFO:teuthology.misc:Compressing logs...
2022-04-06T00:00:01.257 INFO:teuthology.misc:Compressing logs...
2022-04-06T00:00:22.793 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/672c63dc-b506-11ec-8c36-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-04-06T00:00:24.973 INFO:teuthology.orchestra.run.smithi065.stderr:gzip: /var/log/ceph/672c63dc-b506-11ec-8c36-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-04-06T00:00:36.678 INFO:teuthology.orchestra.run.smithi065.stderr:gzip: /var/log/ceph/672c63dc-b506-11ec-8c36-001a4aab830c/ceph-mon.smithi065.log: file size changed while zipping
2022-04-06T00:00:47.012 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/672c63dc-b506-11ec-8c36-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-04-06T00:00:51.411 INFO:teuthology.orchestra.run.smithi065.stderr:gzip: /var/log/ceph/672c63dc-b506-11ec-8c36-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-04-06T00:01:06.041 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/672c63dc-b506-11ec-8c36-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-04-06T00:01:09.299 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/672c63dc-b506-11ec-8c36-001a4aab830c/ceph-mon.smithi073.log: file size changed while zipping
2022-04-06T00:01:14.662 INFO:teuthology.orchestra.run.smithi065.stderr:gzip: /var/log/ceph/672c63dc-b506-11ec-8c36-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-04-06T00:01:27.295 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/672c63dc-b506-11ec-8c36-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-04-06T00:01:34.284 INFO:teuthology.orchestra.run.smithi065.stderr:gzip: /var/log/ceph/672c63dc-b506-11ec-8c36-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-04-06T00:01:36.909 INFO:teuthology.orchestra.run.smithi065.stderr:gzip: /var/log/ceph/672c63dc-b506-11ec-8c36-001a4aab830c/ceph-mgr.smithi065.xokhse.log: file size changed while zipping
2022-04-06T00:02:14.292 INFO:teuthology.orchestra.run.smithi073.stderr:tar: .: file changed as we read it
2022-04-06T00:02:14.353 INFO:teuthology.kill:No teuthology processes running
2022-04-06T00:02:15.481 INFO:teuthology.kill:Nuking machines: ['smithi065', 'smithi073']
2022-04-06T00:02:16.328 INFO:teuthology.kill:2022-04-06 00:02:16,328.328 INFO:teuthology.nuke:targets:
2022-04-06T00:02:16.329 INFO:teuthology.kill:  smithi065.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDErL58gg41hQ6TNxRcuEKtJbEbv3b5tORQzJriGa9WQejYAiSSx6WSKNqMEFkcSpbqWq5BeCACK8ZfBZfjy7fPcbfVDEBDufNxpuUPdT+giUZORtbp6+ZTRuYpP4f7Xs10cnUevrjGJtS0I2BwK5IRTKbIY0udYE0Tn0oyVg5SPzDz6AYuVdeZ+Fksx6OVvBfludDT0pfiO4xAIveiOegNsc0KcCP9DdwSDy44lL+fPFeEilf5F4oEPUpJj3EaSsu/P05gjX+RaHKcm+NuJEXALiMaWJLMWPxTR/BBlIpjDrpku0iT7Ek0ndrTcmlTficIg07OhOaSOJ59HWxwpAsd2CDab5BoSom7PyMgF3GeVuX8QBU4DIL4E1NYliVTEQuWBzKnDHo+eBK6q3uo+XuWfUBybAajlub45dkGehfQ2ekc18dowNE5eCOFMR0XKrD5mp4mXVe9furo1t4zdXhiTHsK14BI0iarz6Q6Xnl7SBz5tvSEhzOrs6yYW1QkBBc=
2022-04-06T00:02:16.329 INFO:teuthology.kill:  smithi073.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCt73jWL7d0soM4v9oscnBFvkJSbKqspHMWpxgtxO5gwk8wZfRI/jQgx0jvy2qcFgAA6UGOYyYybRNtW8gBSmmK1pGE4qMz2EXZ//wxtrRFbT8p52DssENrKzRWh3xFnqc2aCm3F148lVYkBuJ+102RHqPO4YymiRNTEadO1lVrRgUql4Eq8dZhz+mvaEiFO18KXaZns7EuQuX3kOuKOmMCFtWtWbLF1zna9zfr4pE7rZhuG/9tKPSaRW+ulXJ3WczECrS+F4dx4PHrMVLYkujrkAk+Ra97xohTbufCcus4pkSbjdDk7KnfRzvshPWbpY8eHa/Mw2THn5hCnAl7+K8aLUgBqykT4yuP/Aa6NdMXHnQ3oxMrScqwIVWwQKqwb1J7tzVLq6vfkJH3+TPkaCWO+jVvjdUcQpZeY6xxGKrYDTqV9HVBDas5sZh4PhDtGJQsbNJHOanJA+KHwhDXcZpUb65u2Yu+9Q3VLkH4ddp1OHQVkicCBQp3GAulOYgKB1E=
2022-04-06T00:02:16.355 INFO:teuthology.kill:2022-04-06 00:02:16,355.355 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-04-06T00:02:16.357 INFO:teuthology.kill:2022-04-06 00:02:16,356.356 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-04-06T00:02:16.430 INFO:teuthology.kill:2022-04-06 00:02:16,430.430 INFO:teuthology.orchestra.console:Power off smithi073
2022-04-06T00:02:16.442 INFO:teuthology.kill:2022-04-06 00:02:16,441.441 INFO:teuthology.orchestra.console:Power off smithi065
2022-04-06T00:02:24.801 INFO:teuthology.kill:2022-04-06 00:02:24,801.801 INFO:teuthology.orchestra.console:Power off for smithi065 completed
2022-04-06T00:02:24.947 INFO:teuthology.kill:2022-04-06 00:02:24,946.946 INFO:teuthology.lock.ops:unlocked: smithi065.front.sepia.ceph.com
2022-04-06T00:02:28.909 INFO:teuthology.kill:2022-04-06 00:02:28,909.909 INFO:teuthology.orchestra.console:Power off for smithi073 completed
2022-04-06T00:02:29.051 INFO:teuthology.kill:2022-04-06 00:02:29,051.051 INFO:teuthology.lock.ops:unlocked: smithi073.front.sepia.ceph.com
2022-04-06T00:04:29.338 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15