2022-04-07T20:21:01.318 INFO:root:teuthology version: 1.1.0-a347b2ac
2022-04-07T20:21:01.458 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi085.front.sepia.ceph.com' reimaging
2022-04-07T20:21:01.459 INFO:teuthology.lock.ops:Updating [ubuntu@smithi085.front.sepia.ceph.com]: reset os type and version on server
2022-04-07T20:21:01.460 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server
2022-04-07T20:21:01.480 INFO:teuthology.lock.ops:Node 'ubuntu@smithi085.front.sepia.ceph.com' reimaging is complete
2022-04-07T20:21:01.480 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi114.front.sepia.ceph.com' reimaging
2022-04-07T20:21:01.480 INFO:teuthology.lock.ops:Updating [ubuntu@smithi114.front.sepia.ceph.com]: reset os type and version on server
2022-04-07T20:21:01.481 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server
2022-04-07T20:21:01.501 INFO:teuthology.lock.ops:Node 'ubuntu@smithi114.front.sepia.ceph.com' reimaging is complete
2022-04-07T20:21:01.729 INFO:teuthology.provision.fog.smithi114:Scheduling deploy of centos 8.stream
2022-04-07T20:21:01.733 INFO:teuthology.provision.fog.smithi085:Scheduling deploy of centos 8.stream
2022-04-07T20:21:02.083 INFO:teuthology.orchestra.console:Power off smithi085
2022-04-07T20:21:02.148 INFO:teuthology.orchestra.console:Power off smithi114
2022-04-07T20:21:10.508 INFO:teuthology.orchestra.console:Power off for smithi114 completed
2022-04-07T20:21:10.609 INFO:teuthology.orchestra.console:Power on smithi114
2022-04-07T20:21:14.566 INFO:teuthology.orchestra.console:Power off for smithi085 completed
2022-04-07T20:21:14.668 INFO:teuthology.orchestra.console:Power on smithi085
2022-04-07T20:21:18.967 INFO:teuthology.orchestra.console:Power on for smithi114 completed
2022-04-07T20:21:19.068 INFO:teuthology.provision.fog.smithi114:Waiting for deploy to finish
2022-04-07T20:21:23.035 INFO:teuthology.orchestra.console:Power on for smithi085 completed
2022-04-07T20:21:23.137 INFO:teuthology.provision.fog.smithi085:Waiting for deploy to finish
2022-04-07T20:23:50.066 ERROR:teuthology.orchestra.connection:Error authenticating with smithi114.front.sepia.ceph.com: Authentication failed.
2022-04-07T20:25:30.863 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-04-07T20:25:56.631 INFO:teuthology.provision.fog.smithi114:Node is ready
2022-04-07T20:25:56.650 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114.front.sepia.ceph.com
2022-04-07T20:25:56.707 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114 smithi114.front.sepia.ceph.com smithi114
2022-04-07T20:25:57.053 INFO:teuthology.provision.fog.smithi114:Deploy complete!
2022-04-07T20:26:07.014 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-04-07T20:26:07.287 INFO:teuthology.provision.fog.smithi085:Node is ready
2022-04-07T20:26:07.306 INFO:teuthology.orchestra.run.smithi085.stdout:smithi085.front.sepia.ceph.com
2022-04-07T20:26:07.366 INFO:teuthology.orchestra.run.smithi085.stdout:172.21.15.85 smithi085.front.sepia.ceph.com smithi085
2022-04-07T20:26:07.716 INFO:teuthology.provision.fog.smithi085:Deploy complete!
2022-04-07T20:26:07.885 INFO:teuthology.lock.ops:Checking smithi085.front.sepia.ceph.com
2022-04-07T20:26:07.886 INFO:teuthology.lock.ops:New key found. Updating...
2022-04-07T20:26:07.905 INFO:teuthology.lock.ops:Checking smithi114.front.sepia.ceph.com
2022-04-07T20:26:07.905 INFO:teuthology.lock.ops:New key found. Updating...
2022-04-07T20:26:07.924 INFO:teuthology.lock.ops:Updating [smithi085.front.sepia.ceph.com]: set os type and version on server
2022-04-07T20:26:07.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com'
2022-04-07T20:26:08.114 INFO:teuthology.orchestra.run.smithi085.stdout:x86_64
2022-04-07T20:26:08.171 INFO:teuthology.orchestra.run.smithi085.stdout:NAME="CentOS Stream"
2022-04-07T20:26:08.172 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION="8"
2022-04-07T20:26:08.172 INFO:teuthology.orchestra.run.smithi085.stdout:ID="centos"
2022-04-07T20:26:08.172 INFO:teuthology.orchestra.run.smithi085.stdout:ID_LIKE="rhel fedora"
2022-04-07T20:26:08.172 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_ID="8"
2022-04-07T20:26:08.173 INFO:teuthology.orchestra.run.smithi085.stdout:PLATFORM_ID="platform:el8"
2022-04-07T20:26:08.173 INFO:teuthology.orchestra.run.smithi085.stdout:PRETTY_NAME="CentOS Stream 8"
2022-04-07T20:26:08.173 INFO:teuthology.orchestra.run.smithi085.stdout:ANSI_COLOR="0;31"
2022-04-07T20:26:08.173 INFO:teuthology.orchestra.run.smithi085.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-04-07T20:26:08.174 INFO:teuthology.orchestra.run.smithi085.stdout:HOME_URL="https://centos.org/"
2022-04-07T20:26:08.174 INFO:teuthology.orchestra.run.smithi085.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-04-07T20:26:08.174 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-04-07T20:26:08.174 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-04-07T20:26:08.177 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server
2022-04-07T20:26:08.196 INFO:teuthology.lock.ops:Updating [smithi114.front.sepia.ceph.com]: set os type and version on server
2022-04-07T20:26:08.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2022-04-07T20:26:08.394 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64
2022-04-07T20:26:08.453 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream"
2022-04-07T20:26:08.454 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="8"
2022-04-07T20:26:08.454 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos"
2022-04-07T20:26:08.454 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora"
2022-04-07T20:26:08.454 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="8"
2022-04-07T20:26:08.455 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el8"
2022-04-07T20:26:08.455 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 8"
2022-04-07T20:26:08.455 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31"
2022-04-07T20:26:08.455 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-04-07T20:26:08.456 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/"
2022-04-07T20:26:08.456 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-04-07T20:26:08.456 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-04-07T20:26:08.456 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-04-07T20:26:08.457 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server
2022-04-07T20:26:08.559 INFO:teuthology.dispatcher.supervisor:Running job 6781459
2022-04-07T20:26:08.560 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-04-07_20:11:42-orch-wip-yuri2-testing-2022-04-05-1453-pacific-distro-default-smithi/6781459 --name yuriw-2022-04-07_20:11:42-orch-wip-yuri2-testing-2022-04-05-1453-pacific-distro-default-smithi --description orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2022-04-07_20:11:42-orch-wip-yuri2-testing-2022-04-05-1453-pacific-distro-default-smithi/6781459/orig.config.yaml
2022-04-07T20:26:08.563 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-04-07_20:11:42-orch-wip-yuri2-testing-2022-04-05-1453-pacific-distro-default-smithi/6781459
2022-04-07T20:26:08.564 INFO:teuthology.dispatcher.supervisor:Job PID: 18379
2022-04-07T20:26:08.564 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-04-08T02:56:19.272 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-04-08T02:56:19.507 INFO:teuthology.kill:Killing Pids: {18379}
2022-04-08T02:56:20.747 INFO:teuthology.task.internal:roles: ubuntu@smithi085.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-04-08T02:56:20.748 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-04-08T02:56:20.749 INFO:teuthology.misc:Compressing logs...
2022-04-08T02:56:20.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com'
2022-04-08T02:56:20.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2022-04-08T02:56:20.966 INFO:teuthology.orchestra.run.smithi085.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-04-08T02:56:21.178 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-04-08T02:56:21.221 INFO:teuthology.orchestra.run.smithi085.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-04-08T02:56:21.222 INFO:teuthology.orchestra.run.smithi085.stderr:tar: Error is not recoverable: exiting now
2022-04-08T02:56:21.320 INFO:teuthology.orchestra.run.smithi114.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-04-08T02:56:21.321 INFO:teuthology.orchestra.run.smithi114.stderr:tar: Error is not recoverable: exiting now
2022-04-08T02:56:21.322 INFO:teuthology.misc:Compressing logs...
2022-04-08T02:56:21.701 INFO:teuthology.misc:Compressing logs...
2022-04-08T02:56:25.258 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/9400907a-b6b1-11ec-8c36-001a4aab830c/ceph-mon.smithi114.log: file size changed while zipping
2022-04-08T02:56:34.629 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/9400907a-b6b1-11ec-8c36-001a4aab830c/ceph-mon.smithi085.log: file size changed while zipping
2022-04-08T02:56:48.548 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/9400907a-b6b1-11ec-8c36-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-04-08T02:56:59.095 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/9400907a-b6b1-11ec-8c36-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-04-08T02:57:13.487 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/9400907a-b6b1-11ec-8c36-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-04-08T02:57:15.634 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/9400907a-b6b1-11ec-8c36-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-04-08T02:57:35.748 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/9400907a-b6b1-11ec-8c36-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-04-08T02:57:37.349 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/9400907a-b6b1-11ec-8c36-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-04-08T02:57:38.111 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/9400907a-b6b1-11ec-8c36-001a4aab830c/ceph-mgr.smithi114.cnbnbq.log: file size changed while zipping
2022-04-08T02:57:57.633 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/9400907a-b6b1-11ec-8c36-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-04-08T02:57:58.228 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/9400907a-b6b1-11ec-8c36-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-04-08T02:58:18.392 INFO:teuthology.orchestra.run.smithi085.stderr:tar: .: file changed as we read it
2022-04-08T02:58:37.350 INFO:teuthology.kill:No teuthology processes running
2022-04-08T02:58:38.505 INFO:teuthology.kill:Nuking machines: ['smithi085', 'smithi114']
2022-04-08T02:58:39.468 INFO:teuthology.kill:2022-04-08 02:58:39,468.468 INFO:teuthology.nuke:targets:
2022-04-08T02:58:39.469 INFO:teuthology.kill:  smithi085.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmKrh73o3dbgkC7JWwIHM9gKuBEVIqJ6Vy1eRsTJZflvTvoy0x3R5RN7EH0IPfNa7Gn75mc6+WDAJdl2tHuc9TLM0ASFVAqns73tGFDScV72F6krPx5HnCKhwagJIQwRiCbxwEUStmuKVegDH4w+ZdTEehfOy+tTjktwmfuFIGfExAxsdjZcFrS4c9ZbMFYBihDnVGqvf0h1GEVrhEXqfgc7KmqUn6F3RWZrQo28hzym3avOBGhGDnSGsEF6qJchAUVjAP1b+IQ0DdHOxWE7F8tcDtU058f/DKTxz4t924QbFIyeSf0fQMTGkRQ741pYp6ScEXIh8Muj7fOTymJ7DSz2pqEvxg1bVUP5AfNVnan0uQIqnbnDHDf9mUjpPzjmgX3vQVi2S7yfbHBqNbwI4DhUGvDwWd1KXJnNI43p6n/c6kMo9SBWtoNaa5ZpIaJK4NmfwIKfSLIapFS0tItBcmzn6gfPQdGSQrU9HE9FTGvJbMXzVv9CYEGirYaow12mM=
2022-04-08T02:58:39.470 INFO:teuthology.kill:  smithi114.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFRzhY03Rw+WmHA2axlUOzDKiiiZlWPsisDfNoNgO2hleQumYrMkOccWabMdKdccoRtaYDlT2cyHMMRmgjRVvRIxf8mT3xsPVE62LRug5p0k2eAGead925ENivH43maP+K6xglsTZPcsnEDYACehmm6atb6OoZlwxDrvYc4HZehEZbIMJ+NRowM5rs3x6sM7TM2LJffwbfHU7d5AZxqeu51X231e0AoiI31r6HtYVCntUGl0Pj4nS7zJ9qhRxsuP8Fe8fMjQWjPmRbtUw3gTOm1gmCY5h3sFik+ldVvBtm6XiTHzXtwkI7XDXTXSmhunSWrP2iBCM00e5DVWAJjjh/6QxgG0abJa0X2JIh3rcq8IiOym4usW/Ms52V0R1Fd+77V0p5OGuN3gxk31ZgW178AgGkQdSlEtF3CmgIE9QJ6Yw2JuMwfUUQIe/5lihSXWKvhdekLSg0h4UGelFPboSWbTRINe4KTiq2Lt6igsrzpBIyyriH4fBu50uuoIoVfK0=
2022-04-08T02:58:39.496 INFO:teuthology.kill:2022-04-08 02:58:39,496.496 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-04-08T02:58:39.499 INFO:teuthology.kill:2022-04-08 02:58:39,499.499 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-04-08T02:58:39.571 INFO:teuthology.kill:2022-04-08 02:58:39,571.571 INFO:teuthology.orchestra.console:Power off smithi114
2022-04-08T02:58:39.585 INFO:teuthology.kill:2022-04-08 02:58:39,585.585 INFO:teuthology.orchestra.console:Power off smithi085
2022-04-08T02:58:47.951 INFO:teuthology.kill:2022-04-08 02:58:47,950.950 INFO:teuthology.orchestra.console:Power off for smithi085 completed
2022-04-08T02:58:48.098 INFO:teuthology.kill:2022-04-08 02:58:48,097.097 INFO:teuthology.lock.ops:unlocked: smithi085.front.sepia.ceph.com
2022-04-08T02:58:52.059 INFO:teuthology.kill:2022-04-08 02:58:52,058.058 INFO:teuthology.orchestra.console:Power off for smithi114 completed
2022-04-08T02:58:52.207 INFO:teuthology.kill:2022-04-08 02:58:52,207.207 INFO:teuthology.lock.ops:unlocked: smithi114.front.sepia.ceph.com
2022-04-08T03:00:52.444 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-04-08T03:00:52.546 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi085 but it was locked by another job: /home/teuthworker/archive/khiremat-2022-04-08_02:30:12-fs:volumes-wip-khiremat-44347-subvolumegroup-quota-distro-default-smithi/6781950
2022-04-08T03:00:52.546 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi114 but it was locked by another job: /home/teuthworker/archive/khiremat-2022-04-08_02:30:12-fs:volumes-wip-khiremat-44347-subvolumegroup-quota-distro-default-smithi/6781950