2021-11-12T23:00:48.179 INFO:root:teuthology version: 1.1.0-d4737010 2021-11-12T23:00:48.310 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi022.front.sepia.ceph.com' reimaging 2021-11-12T23:00:48.312 INFO:teuthology.lock.ops:Updating [ubuntu@smithi022.front.sepia.ceph.com]: reset os type and version on server 2021-11-12T23:00:48.312 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2021-11-12T23:00:48.333 INFO:teuthology.lock.ops:Node 'ubuntu@smithi022.front.sepia.ceph.com' reimaging is complete 2021-11-12T23:00:48.334 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi197.front.sepia.ceph.com' reimaging 2021-11-12T23:00:48.334 INFO:teuthology.lock.ops:Updating [ubuntu@smithi197.front.sepia.ceph.com]: reset os type and version on server 2021-11-12T23:00:48.335 INFO:teuthology.lock.ops:Updating smithi197.front.sepia.ceph.com on lock server 2021-11-12T23:00:48.355 INFO:teuthology.lock.ops:Node 'ubuntu@smithi197.front.sepia.ceph.com' reimaging is complete 2021-11-12T23:00:48.556 INFO:teuthology.provision.fog.smithi022:Scheduling deploy of centos 8.stream 2021-11-12T23:00:48.600 INFO:teuthology.provision.fog.smithi197:Scheduling deploy of centos 8.stream 2021-11-12T23:00:48.909 INFO:teuthology.orchestra.console:Power off smithi022 2021-11-12T23:00:48.959 INFO:teuthology.orchestra.console:Power off smithi197 2021-11-12T23:01:01.408 INFO:teuthology.orchestra.console:Power off for smithi022 completed 2021-11-12T23:01:01.442 INFO:teuthology.orchestra.console:Power off for smithi197 completed 2021-11-12T23:01:01.509 INFO:teuthology.orchestra.console:Power on smithi022 2021-11-12T23:01:01.544 INFO:teuthology.orchestra.console:Power on smithi197 2021-11-12T23:01:05.799 INFO:teuthology.orchestra.console:Power on for smithi197 completed 2021-11-12T23:01:05.900 INFO:teuthology.provision.fog.smithi197:Waiting for deploy to finish 2021-11-12T23:01:09.879 INFO:teuthology.orchestra.console:Power on for smithi022 completed 2021-11-12T23:01:09.980 INFO:teuthology.provision.fog.smithi022:Waiting for deploy to finish 2021-11-12T23:05:44.415 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-12T23:05:44.679 INFO:teuthology.provision.fog.smithi022:Node is ready 2021-11-12T23:05:44.697 INFO:teuthology.orchestra.run.smithi022.stdout:smithi022.front.sepia.ceph.com 2021-11-12T23:05:44.755 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22 smithi022.front.sepia.ceph.com smithi022 2021-11-12T23:05:45.106 INFO:teuthology.provision.fog.smithi022:Deploy complete! 2021-11-12T23:06:10.572 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-12T23:06:10.877 INFO:teuthology.provision.fog.smithi197:Node is ready 2021-11-12T23:06:10.896 INFO:teuthology.orchestra.run.smithi197.stdout:smithi197.front.sepia.ceph.com 2021-11-12T23:06:10.953 INFO:teuthology.orchestra.run.smithi197.stdout:172.21.15.197 smithi197.front.sepia.ceph.com smithi197 2021-11-12T23:06:11.288 INFO:teuthology.provision.fog.smithi197:Deploy complete! 2021-11-12T23:06:11.439 INFO:teuthology.lock.ops:Checking smithi022.front.sepia.ceph.com 2021-11-12T23:06:11.440 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-12T23:06:11.472 INFO:teuthology.lock.ops:Checking smithi197.front.sepia.ceph.com 2021-11-12T23:06:11.472 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-12T23:06:11.491 INFO:teuthology.lock.ops:Updating [smithi022.front.sepia.ceph.com]: set os type and version on server 2021-11-12T23:06:11.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2021-11-12T23:06:11.776 INFO:teuthology.orchestra.run.smithi022.stdout:x86_64 2021-11-12T23:06:11.835 INFO:teuthology.orchestra.run.smithi022.stdout:NAME="CentOS Stream" 2021-11-12T23:06:11.836 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION="8" 2021-11-12T23:06:11.836 INFO:teuthology.orchestra.run.smithi022.stdout:ID="centos" 2021-11-12T23:06:11.836 INFO:teuthology.orchestra.run.smithi022.stdout:ID_LIKE="rhel fedora" 2021-11-12T23:06:11.837 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION_ID="8" 2021-11-12T23:06:11.837 INFO:teuthology.orchestra.run.smithi022.stdout:PLATFORM_ID="platform:el8" 2021-11-12T23:06:11.837 INFO:teuthology.orchestra.run.smithi022.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-12T23:06:11.837 INFO:teuthology.orchestra.run.smithi022.stdout:ANSI_COLOR="0;31" 2021-11-12T23:06:11.837 INFO:teuthology.orchestra.run.smithi022.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-12T23:06:11.838 INFO:teuthology.orchestra.run.smithi022.stdout:HOME_URL="https://centos.org/" 2021-11-12T23:06:11.838 INFO:teuthology.orchestra.run.smithi022.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-12T23:06:11.838 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-12T23:06:11.838 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-12T23:06:11.840 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2021-11-12T23:06:11.858 INFO:teuthology.lock.ops:Updating [smithi197.front.sepia.ceph.com]: set os type and version on server 2021-11-12T23:06:11.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi197.front.sepia.ceph.com' 2021-11-12T23:06:12.053 INFO:teuthology.orchestra.run.smithi197.stdout:x86_64 2021-11-12T23:06:12.111 INFO:teuthology.orchestra.run.smithi197.stdout:NAME="CentOS Stream" 2021-11-12T23:06:12.112 INFO:teuthology.orchestra.run.smithi197.stdout:VERSION="8" 2021-11-12T23:06:12.112 INFO:teuthology.orchestra.run.smithi197.stdout:ID="centos" 2021-11-12T23:06:12.113 INFO:teuthology.orchestra.run.smithi197.stdout:ID_LIKE="rhel fedora" 2021-11-12T23:06:12.113 INFO:teuthology.orchestra.run.smithi197.stdout:VERSION_ID="8" 2021-11-12T23:06:12.113 INFO:teuthology.orchestra.run.smithi197.stdout:PLATFORM_ID="platform:el8" 2021-11-12T23:06:12.114 INFO:teuthology.orchestra.run.smithi197.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-12T23:06:12.114 INFO:teuthology.orchestra.run.smithi197.stdout:ANSI_COLOR="0;31" 2021-11-12T23:06:12.115 INFO:teuthology.orchestra.run.smithi197.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-12T23:06:12.115 INFO:teuthology.orchestra.run.smithi197.stdout:HOME_URL="https://centos.org/" 2021-11-12T23:06:12.116 INFO:teuthology.orchestra.run.smithi197.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-12T23:06:12.116 INFO:teuthology.orchestra.run.smithi197.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-12T23:06:12.116 INFO:teuthology.orchestra.run.smithi197.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-12T23:06:12.117 INFO:teuthology.lock.ops:Updating smithi197.front.sepia.ceph.com on lock server 2021-11-12T23:06:12.230 INFO:teuthology.dispatcher.supervisor:Running job 6500288 2021-11-12T23:06:12.231 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-11-12_18:55:52-rados-wip-yuri6-testing-2021-11-12-0808-distro-basic-smithi/6500288 --name yuriw-2021-11-12_18:55:52-rados-wip-yuri6-testing-2021-11-12-0808-distro-basic-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2021-11-12_18:55:52-rados-wip-yuri6-testing-2021-11-12-0808-distro-basic-smithi/6500288/orig.config.yaml 2021-11-12T23:06:12.236 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-11-12_18:55:52-rados-wip-yuri6-testing-2021-11-12-0808-distro-basic-smithi/6500288 2021-11-12T23:06:12.237 INFO:teuthology.dispatcher.supervisor:Job PID: 27834 2021-11-12T23:06:12.238 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-11-13T11:07:07.371 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-11-13T11:07:07.589 INFO:teuthology.kill:Killing Pids: {27834} 2021-11-13T11:07:08.795 INFO:teuthology.task.internal:roles: ubuntu@smithi022.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-11-13T11:07:08.796 INFO:teuthology.task.internal:roles: ubuntu@smithi197.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-11-13T11:07:08.797 INFO:teuthology.misc:Compressing logs... 2021-11-13T11:07:08.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2021-11-13T11:07:08.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi197.front.sepia.ceph.com' 2021-11-13T11:07:09.040 INFO:teuthology.orchestra.run.smithi022.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-11-13T11:07:09.233 INFO:teuthology.orchestra.run.smithi197.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-11-13T11:07:09.274 INFO:teuthology.orchestra.run.smithi022.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-11-13T11:07:09.275 INFO:teuthology.orchestra.run.smithi022.stderr:tar: Error is not recoverable: exiting now 2021-11-13T11:07:09.344 INFO:teuthology.orchestra.run.smithi197.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-11-13T11:07:09.344 INFO:teuthology.orchestra.run.smithi197.stderr:tar: Error is not recoverable: exiting now 2021-11-13T11:07:09.345 INFO:teuthology.misc:Compressing logs... 2021-11-13T11:07:09.738 INFO:teuthology.orchestra.run.smithi022.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-11-13T11:07:09.943 INFO:teuthology.misc:Compressing logs... 2021-11-13T11:07:31.318 INFO:teuthology.orchestra.run.smithi197.stderr:gzip: /var/log/ceph/9b1f1e64-440e-11ec-8c2c-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2021-11-13T11:07:36.575 INFO:teuthology.orchestra.run.smithi022.stderr:gzip: /var/log/ceph/9b1f1e64-440e-11ec-8c2c-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2021-11-13T11:07:41.114 INFO:teuthology.orchestra.run.smithi197.stderr:gzip: /var/log/ceph/9b1f1e64-440e-11ec-8c2c-001a4aab830c/ceph-mon.smithi197.log: file size changed while zipping 2021-11-13T11:08:04.586 INFO:teuthology.orchestra.run.smithi022.stderr:gzip: /var/log/ceph/9b1f1e64-440e-11ec-8c2c-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2021-11-13T11:08:05.394 INFO:teuthology.orchestra.run.smithi197.stderr:gzip: /var/log/ceph/9b1f1e64-440e-11ec-8c2c-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2021-11-13T11:08:13.062 INFO:teuthology.orchestra.run.smithi197.stderr:gzip: /var/log/ceph/9b1f1e64-440e-11ec-8c2c-001a4aab830c/ceph-mgr.smithi197.eyxtkw.log: file size changed while zipping 2021-11-13T11:08:22.571 INFO:teuthology.orchestra.run.smithi022.stderr:gzip: /var/log/ceph/9b1f1e64-440e-11ec-8c2c-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2021-11-13T11:08:37.928 INFO:teuthology.orchestra.run.smithi197.stderr:gzip: /var/log/ceph/9b1f1e64-440e-11ec-8c2c-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2021-11-13T11:08:43.529 INFO:teuthology.orchestra.run.smithi022.stderr:gzip: /var/log/ceph/9b1f1e64-440e-11ec-8c2c-001a4aab830c/ceph-mon.smithi022.log: file size changed while zipping 2021-11-13T11:09:07.442 INFO:teuthology.orchestra.run.smithi197.stderr:gzip: /var/log/ceph/9b1f1e64-440e-11ec-8c2c-001a4aab830c/ceph-osd.7.log: file size changed while zipping 2021-11-13T11:09:10.840 INFO:teuthology.orchestra.run.smithi022.stderr:gzip: /var/log/ceph/9b1f1e64-440e-11ec-8c2c-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2021-11-13T11:10:00.510 INFO:teuthology.kill:No teuthology processes running 2021-11-13T11:10:01.862 INFO:teuthology.kill:Nuking machines: ['smithi022', 'smithi197'] 2021-11-13T11:10:02.880 INFO:teuthology.kill:2021-11-13 11:10:02,880.880 INFO:teuthology.nuke:targets: 2021-11-13T11:10:02.881 INFO:teuthology.kill: smithi022.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLysEqsTYH4ymrs0TFDY+k2eTM9wcyENbEKV8CiqpZgqpMzX3P+n7I3xaSUE8oUyzht9cOdIxMoVo23s2SW1esTqf//vPFDeSMAZJGdYGgkxp/z5imMMDY2nyy2Q1CxJIh/MCDOOOrXilTS8wh98Z73Vy4SSh9jspYG1jmWaRiSrgvha0xobJ9OjEKWDzQxjoIy9GHEIxBU8ckJJo5AiWSR/eCYWCl3o90E5RHkFL/6ZzvY143C/TDEkoKRTGE5A4QmisDVlgmWA72JrjVKgQgvoAVLB/8xmDuQv+lCp3gNW9QcUC7PTdU3m6c2IM4mWd60p7kfGW7PYOPWYyHjF4naSAKNYJZN/motUINQ8EUJj1lpIdWakWXUHKbXT4cfO9mFcPYSj+K7tXncj2jlV1DIeLYzVodQynGBAZ8avpFiLQpNJKAEzsm1wYqd88tPkiWW7ex0WibYADIgHHHltSL2+snpGYo8bG/7xQIEq5u1TdFto8YE3U91p6SeEOyZW0= 2021-11-13T11:10:02.881 INFO:teuthology.kill: smithi197.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnkOIUOhXWDeoH7ICv17z9GZNsXzDN9FJ7dr5v/+TpOo62BHUx9VvfSDNO/9bcog+rxX1hwO2hZoX/1PEdMpbK8NaQ6Rdmf1oQ9qg4Eydwl4pgu+9ncTvDm2ULt0z/8siN2yIW9W2SuIWIYyw4RaFHwqF0awQ0gNC0zBATWNbSBrkjplCInaBA58JbX9FL1FrVkk5PNtGbiLePPPLN7UQ/LjSLCh4LjHBZljDjh7qXBVvTbex78CLhb+aURbELVdSgEMpIQHfsXxhOGMph1FOn7p/V8XUFPxkHGAw6c+SNWUF749VYOlMrRckf68sfkewIBTADda6VupH1QBxmavASVfh6ohvfMECoQrb9p0vVBrA/ni850xHcn4HWjfBRNEpYK6pH0wwItFwzO4G3S3s3DBKCuLF0msLCC1kjQx8mL0VXiaCwyWAUV0W2Ju6cOWp+Ss4u9uG4gmTDjRRSEAhOI8GzuVOFflqhq1zAMT84WB+jqL4Obg5hyTMBOZc6pDs= 2021-11-13T11:10:02.909 INFO:teuthology.kill:2021-11-13 11:10:02,909.909 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-13T11:10:02.916 INFO:teuthology.kill:2021-11-13 11:10:02,915.915 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-13T11:10:03.009 INFO:teuthology.kill:2021-11-13 11:10:03,008.008 INFO:teuthology.orchestra.console:Power off smithi022 2021-11-13T11:10:03.050 INFO:teuthology.kill:2021-11-13 11:10:03,050.050 INFO:teuthology.orchestra.console:Power off smithi197 2021-11-13T11:10:11.385 INFO:teuthology.kill:2021-11-13 11:10:11,384.384 INFO:teuthology.orchestra.console:Power off for smithi022 completed 2021-11-13T11:10:11.535 INFO:teuthology.kill:2021-11-13 11:10:11,534.534 INFO:teuthology.lock.ops:unlocked: smithi022.front.sepia.ceph.com 2021-11-13T11:10:15.532 INFO:teuthology.kill:2021-11-13 11:10:15,532.532 INFO:teuthology.orchestra.console:Power off for smithi197 completed 2021-11-13T11:10:15.702 INFO:teuthology.kill:2021-11-13 11:10:15,702.702 INFO:teuthology.lock.ops:unlocked: smithi197.front.sepia.ceph.com 2021-11-13T11:12:16.084 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-11-13T11:12:16.232 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi022 but it was locked by another job: /home/teuthworker/archive/amathuri-2021-11-12_19:23:57-rados-wip-yuri-master-10.29.21-distro-basic-smithi/6500588 2021-11-13T11:12:16.233 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi197 but it was locked by another job: /home/teuthworker/archive/amathuri-2021-11-12_19:23:57-rados-wip-yuri-master-10.29.21-distro-basic-smithi/6500588