2022-01-07T16:42:12.893 INFO:root:teuthology version: 1.1.0-95a7d479
2022-01-07T16:42:13.015 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi114.front.sepia.ceph.com' reimaging
2022-01-07T16:42:13.016 INFO:teuthology.lock.ops:Updating [ubuntu@smithi114.front.sepia.ceph.com]: reset os type and version on server
2022-01-07T16:42:13.017 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server
2022-01-07T16:42:13.035 INFO:teuthology.lock.ops:Node 'ubuntu@smithi114.front.sepia.ceph.com' reimaging is complete
2022-01-07T16:42:13.035 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi161.front.sepia.ceph.com' reimaging
2022-01-07T16:42:13.036 INFO:teuthology.lock.ops:Updating [ubuntu@smithi161.front.sepia.ceph.com]: reset os type and version on server
2022-01-07T16:42:13.036 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server
2022-01-07T16:42:13.055 INFO:teuthology.lock.ops:Node 'ubuntu@smithi161.front.sepia.ceph.com' reimaging is complete
2022-01-07T16:42:13.255 INFO:teuthology.provision.fog.smithi114:Scheduling deploy of centos 8.stream
2022-01-07T16:42:13.264 INFO:teuthology.provision.fog.smithi161:Scheduling deploy of centos 8.stream
2022-01-07T16:42:13.631 INFO:teuthology.orchestra.console:Power off smithi161
2022-01-07T16:42:13.647 INFO:teuthology.orchestra.console:Power off smithi114
2022-01-07T16:42:13.775 INFO:teuthology.orchestra.console:Power off for smithi161 completed
2022-01-07T16:42:13.876 INFO:teuthology.orchestra.console:Power on smithi161
2022-01-07T16:42:18.126 INFO:teuthology.orchestra.console:Power on for smithi161 completed
2022-01-07T16:42:18.228 INFO:teuthology.provision.fog.smithi161:Waiting for deploy to finish
2022-01-07T16:42:22.010 INFO:teuthology.orchestra.console:Power off for smithi114 completed
2022-01-07T16:42:22.112 INFO:teuthology.orchestra.console:Power on smithi114
2022-01-07T16:42:30.479 INFO:teuthology.orchestra.console:Power on for smithi114 completed
2022-01-07T16:42:30.580 INFO:teuthology.provision.fog.smithi114:Waiting for deploy to finish
2022-01-07T16:47:20.111 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-01-07T16:47:20.384 INFO:teuthology.provision.fog.smithi114:Node is ready
2022-01-07T16:47:20.401 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114
2022-01-07T16:47:20.457 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114 smithi114.front.sepia.ceph.com smithi114
2022-01-07T16:47:20.791 INFO:teuthology.provision.fog.smithi114:Deploy complete!
2022-01-07T16:47:34.669 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-01-07T16:47:34.938 INFO:teuthology.provision.fog.smithi161:Node is ready
2022-01-07T16:47:34.955 INFO:teuthology.orchestra.run.smithi161.stdout:smithi161.front.sepia.ceph.com
2022-01-07T16:47:35.012 INFO:teuthology.orchestra.run.smithi161.stdout:172.21.15.161 smithi161.front.sepia.ceph.com smithi161
2022-01-07T16:47:35.345 INFO:teuthology.provision.fog.smithi161:Deploy complete!
2022-01-07T16:47:35.525 INFO:teuthology.lock.ops:Checking smithi114.front.sepia.ceph.com
2022-01-07T16:47:35.526 INFO:teuthology.lock.ops:New key found. Updating...
2022-01-07T16:47:35.546 INFO:teuthology.lock.ops:Checking smithi161.front.sepia.ceph.com
2022-01-07T16:47:35.546 INFO:teuthology.lock.ops:New key found. Updating...
2022-01-07T16:47:35.563 INFO:teuthology.lock.ops:Updating [smithi114.front.sepia.ceph.com]: set os type and version on server
2022-01-07T16:47:35.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2022-01-07T16:47:35.803 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64
2022-01-07T16:47:35.859 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream"
2022-01-07T16:47:35.859 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="8"
2022-01-07T16:47:35.859 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos"
2022-01-07T16:47:35.860 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora"
2022-01-07T16:47:35.860 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="8"
2022-01-07T16:47:35.860 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el8"
2022-01-07T16:47:35.860 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 8"
2022-01-07T16:47:35.861 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31"
2022-01-07T16:47:35.861 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-01-07T16:47:35.861 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/"
2022-01-07T16:47:35.861 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-01-07T16:47:35.861 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-01-07T16:47:35.862 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-01-07T16:47:35.863 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server
2022-01-07T16:47:35.883 INFO:teuthology.lock.ops:Updating [smithi161.front.sepia.ceph.com]: set os type and version on server
2022-01-07T16:47:35.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com'
2022-01-07T16:47:36.070 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64
2022-01-07T16:47:36.126 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Stream"
2022-01-07T16:47:36.126 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="8"
2022-01-07T16:47:36.126 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos"
2022-01-07T16:47:36.126 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora"
2022-01-07T16:47:36.127 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="8"
2022-01-07T16:47:36.127 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el8"
2022-01-07T16:47:36.127 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Stream 8"
2022-01-07T16:47:36.127 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31"
2022-01-07T16:47:36.128 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-01-07T16:47:36.128 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://centos.org/"
2022-01-07T16:47:36.128 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-01-07T16:47:36.128 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-01-07T16:47:36.129 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-01-07T16:47:36.129 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server
2022-01-07T16:47:36.236 INFO:teuthology.dispatcher.supervisor:Running job 6598681
2022-01-07T16:47:36.236 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-01-06_14:42:22-rados-WIP_GBH_safe_fast_shutdown_A3-distro-basic-smithi/6598681 --name benhanokh-2022-01-06_14:42:22-rados-WIP_GBH_safe_fast_shutdown_A3-distro-basic-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/benhanokh-2022-01-06_14:42:22-rados-WIP_GBH_safe_fast_shutdown_A3-distro-basic-smithi/6598681/orig.config.yaml
2022-01-07T16:47:36.240 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-01-06_14:42:22-rados-WIP_GBH_safe_fast_shutdown_A3-distro-basic-smithi/6598681
2022-01-07T16:47:36.241 INFO:teuthology.dispatcher.supervisor:Job PID: 19649
2022-01-07T16:47:36.241 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-01-08T04:48:31.245 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2022-01-08T04:48:31.473 INFO:teuthology.kill:Killing Pids: {19649}
2022-01-08T04:48:32.830 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-01-08T04:48:32.831 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-01-08T04:48:32.831 INFO:teuthology.misc:Compressing logs...
2022-01-08T04:48:32.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2022-01-08T04:48:33.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com'
2022-01-08T04:48:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-01-08T04:48:33.269 INFO:teuthology.orchestra.run.smithi161.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-01-08T04:48:33.315 INFO:teuthology.orchestra.run.smithi114.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-01-08T04:48:33.315 INFO:teuthology.orchestra.run.smithi114.stderr:tar: Error is not recoverable: exiting now
2022-01-08T04:48:33.354 INFO:teuthology.orchestra.run.smithi161.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-01-08T04:48:33.355 INFO:teuthology.orchestra.run.smithi161.stderr:tar: Error is not recoverable: exiting now
2022-01-08T04:48:33.360 INFO:teuthology.misc:Compressing logs...
2022-01-08T04:48:33.643 INFO:teuthology.orchestra.run.smithi161.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-01-08T04:48:33.720 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-01-08T04:48:33.919 INFO:teuthology.misc:Compressing logs...
2022-01-08T04:48:34.533 INFO:teuthology.orchestra.run.smithi161.stderr:gzip: /var/log/ceph/abcf8710-6fda-11ec-8c32-001a4aab830c/ceph-mgr.smithi161.ikedpd.log: file size changed while zipping
2022-01-08T04:48:36.102 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/abcf8710-6fda-11ec-8c32-001a4aab830c/ceph-mon.smithi114.log: file size changed while zipping
2022-01-08T04:48:37.089 INFO:teuthology.orchestra.run.smithi161.stderr:gzip: /var/log/ceph/abcf8710-6fda-11ec-8c32-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-01-08T04:48:38.219 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/abcf8710-6fda-11ec-8c32-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-01-08T04:48:39.290 INFO:teuthology.orchestra.run.smithi161.stderr:gzip: /var/log/ceph/abcf8710-6fda-11ec-8c32-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-01-08T04:48:40.232 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/abcf8710-6fda-11ec-8c32-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-01-08T04:48:41.723 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/abcf8710-6fda-11ec-8c32-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-01-08T04:48:41.837 INFO:teuthology.orchestra.run.smithi161.stderr:gzip: /var/log/ceph/abcf8710-6fda-11ec-8c32-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-01-08T04:48:43.921 INFO:teuthology.orchestra.run.smithi161.stderr:gzip: /var/log/ceph/abcf8710-6fda-11ec-8c32-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-01-08T04:48:43.961 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/abcf8710-6fda-11ec-8c32-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-01-08T04:48:45.194 INFO:teuthology.orchestra.run.smithi161.stderr:gzip: /var/log/ceph/abcf8710-6fda-11ec-8c32-001a4aab830c/ceph-mon.smithi161.log: file size changed while zipping
2022-01-08T04:49:33.908 INFO:teuthology.kill:No teuthology processes running
2022-01-08T04:49:35.101 INFO:teuthology.kill:Nuking machines: ['smithi114', 'smithi161']
2022-01-08T04:49:36.165 INFO:teuthology.kill:2022-01-08 04:49:36,165.165 INFO:teuthology.nuke:targets:
2022-01-08T04:49:36.166 INFO:teuthology.kill:  smithi114.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtcw19T1f6rhqqYCF6RR3J5FQ7jMDQ/TZSGBFSoEO1CqxOAy2ZYxsyMrgPB81Uv0jbOd84p0PmcsWVUL2Y7uzvBFvMOqxC+MY7tcl2NpdUfaAnV93ryKh0jnEB7w8WXcU7wV4LUQ9BbD8ELOHMAd2okjJQEqcSIot48RgdCsUlGvrgV5GuToeY/PlPbhBhZq5LimJpW8nVRImw43Kc/WGo2EU7PkfUgB1fIbjZNQDOjHi1HlvCNgmZr6PaRvEQq2CTr7SzZk2DVHRJ6RyionDhc3aouey3zU6zjW1R1+R5Mtg94gETqK70ErIkbfyA2Ir//q866jshkamJxwKdkR6GtxEV7P430qAUsrLxH40ALEM9kdjyMNzqO9hJvyWp3sqOpgH/oMJ6amt1J1y1S0EbW/LkYT4RbrFGAbjtKFXHQ5dX77ECsHuQsu9y79rTF3nmXDCTRjC68BL7oVi5bTTtmuM3GmWsamAc/qm7bDgDfLLtTrk8yYszs40gChXF1u0=
2022-01-08T04:49:36.166 INFO:teuthology.kill:  smithi161.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPnNkNSUPAzQ6a3RbGPWO//NvMxewFaCPYUV6AnNl3P4jXeReE7s35TDFkC4ZO92FF+ApxKe/GGMfz4FS6n+H2BeCPiqheKK5zcpd3ezLcIeLsNWHWBZgELcv7K47i9lrHHIRHI3qwci4FTKa3Ol0D0U5OM7pV/EDPrKJ6KZtrVrpDSOLgjgIyk+U8l2sX48S1U/WPicLPdPxWqs2YbxHtOdYSuVuxWvPtXy1+wAi5kgdT5ov9d1Pp2P3+LNEcF0wG5Hgtxxp5e+vdYPxfiTfnrEGHXd5CdjfYJJb9jof2KJ525ybuZPsCIKBkqCYvVG9k/J5vhSN1ilxcBleOwrpKAZ0XlaXj1FYnIAUWohrmD26fMm7HKhuFnwUE+BwEMRhykOLCgoboGFhAqJ+4ThhA8EDsnx8egG3GGkRzW42ZIVng4mloeUeQOMq0zVBgCRDkJIKPUYN5dJ3Xzoi5p4ejNCYJmfGMljK3vJHWfOaRo5ng8+gA3WDqR9hvi0UCu/0=
2022-01-08T04:49:36.194 INFO:teuthology.kill:2022-01-08 04:49:36,194.194 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-01-08T04:49:36.198 INFO:teuthology.kill:2022-01-08 04:49:36,197.197 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-01-08T04:49:36.277 INFO:teuthology.kill:2022-01-08 04:49:36,277.277 INFO:teuthology.orchestra.console:Power off smithi114
2022-01-08T04:49:36.300 INFO:teuthology.kill:2022-01-08 04:49:36,299.299 INFO:teuthology.orchestra.console:Power off smithi161
2022-01-08T04:49:44.678 INFO:teuthology.kill:2022-01-08 04:49:44,678.678 INFO:teuthology.orchestra.console:Power off for smithi114 completed
2022-01-08T04:49:44.688 INFO:teuthology.kill:2022-01-08 04:49:44,688.688 INFO:teuthology.orchestra.console:Power off for smithi161 completed
2022-01-08T04:49:44.829 INFO:teuthology.kill:2022-01-08 04:49:44,828.828 INFO:teuthology.lock.ops:unlocked: smithi114.front.sepia.ceph.com
2022-01-08T04:49:44.833 INFO:teuthology.kill:2022-01-08 04:49:44,833.833 INFO:teuthology.lock.ops:unlocked: smithi161.front.sepia.ceph.com
2022-01-08T04:51:45.167 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-01-08T04:51:45.270 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi114 but it was locked by another job: /home/teuthworker/archive/akupczyk-2022-01-07_13:16:17-rados-wip-bluefs-fine-grain-locking-4-debug-mutex-on-distro-default-smithi/6601147
2022-01-08T04:51:45.271 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi161 but it was locked by another job: /home/teuthworker/archive/akupczyk-2022-01-07_13:16:17-rados-wip-bluefs-fine-grain-locking-4-debug-mutex-on-distro-default-smithi/6601146