2021-12-15T12:53:03.943 INFO:root:teuthology version: 1.1.0-95a7d479
2021-12-15T12:53:04.077 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi192.front.sepia.ceph.com' reimaging
2021-12-15T12:53:04.078 INFO:teuthology.lock.ops:Updating [ubuntu@smithi192.front.sepia.ceph.com]: reset os type and version on server
2021-12-15T12:53:04.079 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server
2021-12-15T12:53:04.098 INFO:teuthology.lock.ops:Node 'ubuntu@smithi192.front.sepia.ceph.com' reimaging is complete
2021-12-15T12:53:04.098 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi194.front.sepia.ceph.com' reimaging
2021-12-15T12:53:04.098 INFO:teuthology.lock.ops:Updating [ubuntu@smithi194.front.sepia.ceph.com]: reset os type and version on server
2021-12-15T12:53:04.099 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server
2021-12-15T12:53:04.118 INFO:teuthology.lock.ops:Node 'ubuntu@smithi194.front.sepia.ceph.com' reimaging is complete
2021-12-15T12:53:04.310 INFO:teuthology.provision.fog.smithi192:Scheduling deploy of centos 8.3
2021-12-15T12:53:04.402 INFO:teuthology.provision.fog.smithi194:Scheduling deploy of centos 8.3
2021-12-15T12:53:04.646 INFO:teuthology.orchestra.console:Power off smithi192
2021-12-15T12:53:04.777 INFO:teuthology.orchestra.console:Power off smithi194
2021-12-15T12:53:13.013 INFO:teuthology.orchestra.console:Power off for smithi192 completed
2021-12-15T12:53:13.115 INFO:teuthology.orchestra.console:Power on smithi192
2021-12-15T12:53:17.249 INFO:teuthology.orchestra.console:Power off for smithi194 completed
2021-12-15T12:53:17.352 INFO:teuthology.orchestra.console:Power on smithi194
2021-12-15T12:53:21.479 INFO:teuthology.orchestra.console:Power on for smithi192 completed
2021-12-15T12:53:21.580 INFO:teuthology.provision.fog.smithi192:Waiting for deploy to finish
2021-12-15T12:53:21.604 INFO:teuthology.orchestra.console:Power on for smithi194 completed
2021-12-15T12:53:21.705 INFO:teuthology.provision.fog.smithi194:Waiting for deploy to finish
2021-12-15T12:57:36.698 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-12-15T12:57:52.065 INFO:teuthology.provision.fog.smithi194:Node is ready
2021-12-15T12:57:52.084 INFO:teuthology.orchestra.run.smithi194.stdout:smithi194.front.sepia.ceph.com
2021-12-15T12:57:52.140 INFO:teuthology.orchestra.run.smithi194.stdout:172.21.15.194 smithi194.front.sepia.ceph.com smithi194
2021-12-15T12:57:52.510 INFO:teuthology.provision.fog.smithi194:Deploy complete!
2021-12-15T13:01:17.886 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-12-15T13:01:18.192 INFO:teuthology.provision.fog.smithi192:Node is ready
2021-12-15T13:01:18.208 INFO:teuthology.orchestra.run.smithi192.stdout:smithi192.front.sepia.ceph.com
2021-12-15T13:01:18.270 INFO:teuthology.orchestra.run.smithi192.stdout:172.21.15.192 smithi192.front.sepia.ceph.com smithi192
2021-12-15T13:01:18.588 INFO:teuthology.provision.fog.smithi192:Deploy complete!
2021-12-15T13:01:18.764 INFO:teuthology.lock.ops:Checking smithi192.front.sepia.ceph.com
2021-12-15T13:01:18.765 INFO:teuthology.lock.ops:New key found. Updating...
2021-12-15T13:01:18.785 INFO:teuthology.lock.ops:Checking smithi194.front.sepia.ceph.com
2021-12-15T13:01:18.785 INFO:teuthology.lock.ops:New key found. Updating...
2021-12-15T13:01:18.807 INFO:teuthology.lock.ops:Updating [smithi192.front.sepia.ceph.com]: set os type and version on server
2021-12-15T13:01:18.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com'
2021-12-15T13:01:19.074 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64
2021-12-15T13:01:19.128 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Linux"
2021-12-15T13:01:19.129 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="8"
2021-12-15T13:01:19.129 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos"
2021-12-15T13:01:19.129 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora"
2021-12-15T13:01:19.130 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="8"
2021-12-15T13:01:19.130 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el8"
2021-12-15T13:01:19.130 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Linux 8"
2021-12-15T13:01:19.130 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31"
2021-12-15T13:01:19.131 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-12-15T13:01:19.131 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/"
2021-12-15T13:01:19.131 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-12-15T13:01:19.131 INFO:teuthology.orchestra.run.smithi192.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-12-15T13:01:19.132 INFO:teuthology.orchestra.run.smithi192.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-12-15T13:01:19.133 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server
2021-12-15T13:01:19.155 INFO:teuthology.lock.ops:Updating [smithi194.front.sepia.ceph.com]: set os type and version on server
2021-12-15T13:01:19.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com'
2021-12-15T13:01:19.382 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64
2021-12-15T13:01:19.439 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Linux"
2021-12-15T13:01:19.440 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="8"
2021-12-15T13:01:19.440 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos"
2021-12-15T13:01:19.440 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora"
2021-12-15T13:01:19.440 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="8"
2021-12-15T13:01:19.441 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el8"
2021-12-15T13:01:19.441 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Linux 8"
2021-12-15T13:01:19.441 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31"
2021-12-15T13:01:19.443 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-12-15T13:01:19.443 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/"
2021-12-15T13:01:19.443 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-12-15T13:01:19.443 INFO:teuthology.orchestra.run.smithi194.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-12-15T13:01:19.444 INFO:teuthology.orchestra.run.smithi194.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-12-15T13:01:19.444 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server
2021-12-15T13:01:19.539 INFO:teuthology.dispatcher.supervisor:Running job 6563951
2021-12-15T13:01:19.539 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology -v --owner scheduled_swagner@teuthology --archive /home/teuthworker/archive/swagner-2021-12-15_12:52:12-rados:cephadm-wip-swagner2-testing-2021-12-15-1027-pacific-distro-default-smithi/6563951 --name swagner-2021-12-15_12:52:12-rados:cephadm-wip-swagner2-testing-2021-12-15-1027-pacific-distro-default-smithi --description rados:cephadm/mgr-nfs-upgrade/{0-distro/centos_8.3_container_tools_3.0 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/swagner-2021-12-15_12:52:12-rados:cephadm-wip-swagner2-testing-2021-12-15-1027-pacific-distro-default-smithi/6563951/orig.config.yaml
2021-12-15T13:01:19.543 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/swagner-2021-12-15_12:52:12-rados:cephadm-wip-swagner2-testing-2021-12-15-1027-pacific-distro-default-smithi/6563951
2021-12-15T13:01:19.544 INFO:teuthology.dispatcher.supervisor:Job PID: 31659
2021-12-15T13:01:19.544 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-12-16T01:02:13.247 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-12-16T01:02:13.444 INFO:teuthology.kill:Killing Pids: {31659}
2021-12-16T01:02:17.061 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2021-12-16T01:02:17.062 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2021-12-16T01:02:17.063 INFO:teuthology.misc:Compressing logs...
2021-12-16T01:02:17.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com'
2021-12-16T01:02:17.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com'
2021-12-16T01:02:17.324 INFO:teuthology.orchestra.run.smithi192.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2021-12-16T01:02:17.575 INFO:teuthology.orchestra.run.smithi194.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2021-12-16T01:02:17.629 INFO:teuthology.orchestra.run.smithi192.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-12-16T01:02:17.630 INFO:teuthology.orchestra.run.smithi192.stderr:tar: Error is not recoverable: exiting now
2021-12-16T01:02:17.665 INFO:teuthology.orchestra.run.smithi194.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-12-16T01:02:17.666 INFO:teuthology.orchestra.run.smithi194.stderr:tar: Error is not recoverable: exiting now
2021-12-16T01:02:17.667 INFO:teuthology.misc:Compressing logs...
2021-12-16T01:02:17.838 INFO:teuthology.orchestra.run.smithi192.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-12-16T01:02:18.070 INFO:teuthology.misc:Compressing logs...
2021-12-16T01:02:51.139 INFO:teuthology.orchestra.run.smithi192.stderr:gzip: /var/log/ceph/90a177f8-5da8-11ec-8c31-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2021-12-16T01:02:53.119 INFO:teuthology.orchestra.run.smithi194.stderr:gzip: /var/log/ceph/90a177f8-5da8-11ec-8c31-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2021-12-16T01:03:19.667 INFO:teuthology.orchestra.run.smithi192.stderr:gzip: /var/log/ceph/90a177f8-5da8-11ec-8c31-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2021-12-16T01:03:24.282 INFO:teuthology.orchestra.run.smithi194.stderr:gzip: /var/log/ceph/90a177f8-5da8-11ec-8c31-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2021-12-16T01:03:30.432 INFO:teuthology.orchestra.run.smithi194.stderr:gzip: /var/log/ceph/90a177f8-5da8-11ec-8c31-001a4aab830c/ceph-mon.smithi194.log: file size changed while zipping
2021-12-16T01:03:42.622 INFO:teuthology.orchestra.run.smithi192.stderr:gzip: /var/log/ceph/90a177f8-5da8-11ec-8c31-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2021-12-16T01:03:59.414 INFO:teuthology.orchestra.run.smithi194.stderr:gzip: /var/log/ceph/90a177f8-5da8-11ec-8c31-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2021-12-16T01:04:02.843 INFO:teuthology.orchestra.run.smithi192.stderr:gzip: /var/log/ceph/90a177f8-5da8-11ec-8c31-001a4aab830c/ceph-mon.smithi192.log: file size changed while zipping
2021-12-16T01:04:26.075 INFO:teuthology.orchestra.run.smithi194.stderr:gzip: /var/log/ceph/90a177f8-5da8-11ec-8c31-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2021-12-16T01:04:35.357 INFO:teuthology.orchestra.run.smithi192.stderr:gzip: /var/log/ceph/90a177f8-5da8-11ec-8c31-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2021-12-16T01:04:43.307 INFO:teuthology.orchestra.run.smithi192.stderr:gzip: /var/log/ceph/90a177f8-5da8-11ec-8c31-001a4aab830c/ceph-mgr.smithi192.axnctn.log: file size changed while zipping
2021-12-16T01:05:49.081 INFO:teuthology.orchestra.run.smithi192.stderr:tar: .: file changed as we read it
2021-12-16T01:06:52.076 INFO:teuthology.kill:No teuthology processes running
2021-12-16T01:06:54.865 INFO:teuthology.kill:Nuking machines: ['smithi192', 'smithi194']
2021-12-16T01:06:57.646 INFO:teuthology.kill:2021-12-16 01:06:57,644.644 INFO:teuthology.nuke:targets:
2021-12-16T01:06:57.671 INFO:teuthology.kill:  smithi192.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7K36Z7Fgid0FGbIHpH/rMvwY6/MndNRkLrbU9o6qJwWS6xf6sQ4D7qXERY44fe1nQ7X3GCtsZwOnliAT0v8oOk8jqfH7ZDaJ5+QWfW5HZxOEyQvMXrNzsi8PsRH+kLms+5Vx/NdF3dlgJj2DSF+xSDV8ABF1a/KEvH0VpWR5L8OKwfJGrbxILx2XuWENFDFcd+K/Cl3Q2CsG/YTC2gVQoFz10Ar8AK3BvLTeKYWrod9q40ZVz2ZzqkYsU1aHASjUpeQ6A2mokxdM6sPh/Zw6I/1wSIs90OclMDMG1ZxCLPHSjUmiWywqrfxT6BPsM1PNl2ZSBYwpy22dr7vWcxYkZvFELNmc7I9+sIKc7JB3CrTKw1SQze0pK8EHc1rq/NE6oz+antfHqTJJYhDi46nY7oU2Cs4PlPVLRqlAfihzt8Kp/c3e+5yxRBFKk6ua2tS4JYNia9bU2Wn0lD+sLBi5WOp3oxS8TJSGTIcvFtoh5AWpuDlpaeuwqpqgFFq3tR28=
2021-12-16T01:06:57.672 INFO:teuthology.kill:  smithi194.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCy4UjBnLeKqbqvnMKg+Y9NQqH4c85Yh5VK9sHuNSFyGNsOvqsXaVEnVpdpIy5jCDZTUuQ2cc0d/dJl+CUIylbhAamL4vg7kytuX41hhV7IlZAltV+5NJ/U2WgVRVYNV+lA+KfpQvGDIdVbyPKPOVeGpUoUOPuWjadSslsXi1Sv2Qiyop9fPRMSbEh5TMHQtCz8cMt5GW2Y5vNPYT7laPYSahSM2ALPZ0adD8iyGTsNu+/YSsV4+MssJTn0Dko9ZE2SAyb/I5Kpr00dg6XD5E0/tPaO/if4Yuni+Fxg4kLzXY0H1RNX1qeRWKpHcfVUYc35nD+FsvKvF+RnXgq26Rh8Mz7p14xcg/RaeMZBl9tFPg4ygJnYilovleiO0IwUhQ5Ab+dvA8f3rkbu7Q6ivPnQqTUvW0MSJk/nvzL/cX0bqVB54O8sfgnwcvGyrxvIHDXIXNXCZqxQB5WIcxem1yidqGI2MRwycEbnns8npKOVHvqMOSJsJ7rBCAQWHHwdrGU=
2021-12-16T01:06:57.686 INFO:teuthology.kill:2021-12-16 01:06:57,686.686 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-12-16T01:06:57.690 INFO:teuthology.kill:2021-12-16 01:06:57,690.690 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-12-16T01:06:57.803 INFO:teuthology.kill:2021-12-16 01:06:57,802.802 INFO:teuthology.orchestra.console:Power off smithi192
2021-12-16T01:06:57.911 INFO:teuthology.kill:2021-12-16 01:06:57,910.910 INFO:teuthology.orchestra.console:Power off smithi194
2021-12-16T01:07:06.386 INFO:teuthology.kill:2021-12-16 01:07:06,385.385 INFO:teuthology.orchestra.console:Power off for smithi192 completed
2021-12-16T01:07:06.465 INFO:teuthology.kill:2021-12-16 01:07:06,465.465 INFO:teuthology.orchestra.console:Power off for smithi194 completed
2021-12-16T01:07:06.540 INFO:teuthology.kill:2021-12-16 01:07:06,539.539 INFO:teuthology.lock.ops:unlocked: smithi192.front.sepia.ceph.com
2021-12-16T01:07:06.618 INFO:teuthology.kill:2021-12-16 01:07:06,612.612 INFO:teuthology.lock.ops:unlocked: smithi194.front.sepia.ceph.com
2021-12-16T01:09:07.401 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-12-16T01:09:07.561 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi194 but it was locked by another job: /home/teuthworker/archive/akupczyk-2021-12-15_18:10:21-rados-wip-bluefs-fine-grain-locking-4-rebase-distro-default-smithi/6564929