2021-11-26T17:21:53.146 INFO:root:teuthology version: 1.1.0-a0bc3188
2021-11-26T17:21:53.271 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi137.front.sepia.ceph.com' reimaging
2021-11-26T17:21:53.272 INFO:teuthology.lock.ops:Updating [ubuntu@smithi137.front.sepia.ceph.com]: reset os type and version on server
2021-11-26T17:21:53.272 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server
2021-11-26T17:21:53.292 INFO:teuthology.lock.ops:Node 'ubuntu@smithi137.front.sepia.ceph.com' reimaging is complete
2021-11-26T17:21:53.292 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi143.front.sepia.ceph.com' reimaging
2021-11-26T17:21:53.292 INFO:teuthology.lock.ops:Updating [ubuntu@smithi143.front.sepia.ceph.com]: reset os type and version on server
2021-11-26T17:21:53.293 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server
2021-11-26T17:21:53.312 INFO:teuthology.lock.ops:Node 'ubuntu@smithi143.front.sepia.ceph.com' reimaging is complete
2021-11-26T17:21:53.519 INFO:teuthology.provision.fog.smithi137:Scheduling deploy of centos 8.stream
2021-11-26T17:21:53.526 INFO:teuthology.provision.fog.smithi143:Scheduling deploy of centos 8.stream
2021-11-26T17:21:53.877 INFO:teuthology.orchestra.console:Power off smithi137
2021-11-26T17:21:53.892 INFO:teuthology.orchestra.console:Power off smithi143
2021-11-26T17:22:02.250 INFO:teuthology.orchestra.console:Power off for smithi137 completed
2021-11-26T17:22:02.352 INFO:teuthology.orchestra.console:Power on smithi137
2021-11-26T17:22:06.369 INFO:teuthology.orchestra.console:Power off for smithi143 completed
2021-11-26T17:22:06.470 INFO:teuthology.orchestra.console:Power on smithi143
2021-11-26T17:22:10.720 INFO:teuthology.orchestra.console:Power on for smithi137 completed
2021-11-26T17:22:10.726 INFO:teuthology.orchestra.console:Power on for smithi143 completed
2021-11-26T17:22:10.822 INFO:teuthology.provision.fog.smithi137:Waiting for deploy to finish
2021-11-26T17:22:10.828 INFO:teuthology.provision.fog.smithi143:Waiting for deploy to finish
2021-11-26T17:27:00.818 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-26T17:27:00.904 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-26T17:27:01.091 INFO:teuthology.provision.fog.smithi137:Node is ready
2021-11-26T17:27:01.148 INFO:teuthology.orchestra.run.smithi137.stdout:smithi137.front.sepia.ceph.com
2021-11-26T17:27:01.173 INFO:teuthology.provision.fog.smithi143:Node is ready
2021-11-26T17:27:01.190 INFO:teuthology.orchestra.run.smithi143.stdout:smithi143.front.sepia.ceph.com
2021-11-26T17:27:01.208 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.15.137 smithi137.front.sepia.ceph.com smithi137
2021-11-26T17:27:01.247 INFO:teuthology.orchestra.run.smithi143.stdout:172.21.15.143 smithi143.front.sepia.ceph.com smithi143
2021-11-26T17:27:01.560 INFO:teuthology.provision.fog.smithi137:Deploy complete!
2021-11-26T17:27:01.585 INFO:teuthology.provision.fog.smithi143:Deploy complete!
2021-11-26T17:27:01.746 INFO:teuthology.lock.ops:Checking smithi137.front.sepia.ceph.com
2021-11-26T17:27:01.747 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-26T17:27:01.766 INFO:teuthology.lock.ops:Checking smithi143.front.sepia.ceph.com
2021-11-26T17:27:01.766 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-26T17:27:01.786 INFO:teuthology.lock.ops:Updating [smithi137.front.sepia.ceph.com]: set os type and version on server
2021-11-26T17:27:01.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com'
2021-11-26T17:27:02.022 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64
2021-11-26T17:27:02.078 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream"
2021-11-26T17:27:02.078 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="8"
2021-11-26T17:27:02.079 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos"
2021-11-26T17:27:02.079 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora"
2021-11-26T17:27:02.080 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="8"
2021-11-26T17:27:02.080 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el8"
2021-11-26T17:27:02.081 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 8"
2021-11-26T17:27:02.081 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31"
2021-11-26T17:27:02.081 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-26T17:27:02.081 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/"
2021-11-26T17:27:02.081 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-11-26T17:27:02.082 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2021-11-26T17:27:02.082 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2021-11-26T17:27:02.084 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server
2021-11-26T17:27:02.103 INFO:teuthology.lock.ops:Updating [smithi143.front.sepia.ceph.com]: set os type and version on server
2021-11-26T17:27:02.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com'
2021-11-26T17:27:02.324 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64
2021-11-26T17:27:02.379 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Stream"
2021-11-26T17:27:02.380 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="8"
2021-11-26T17:27:02.380 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos"
2021-11-26T17:27:02.380 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora"
2021-11-26T17:27:02.380 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="8"
2021-11-26T17:27:02.381 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el8"
2021-11-26T17:27:02.381 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Stream 8"
2021-11-26T17:27:02.381 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31"
2021-11-26T17:27:02.381 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-26T17:27:02.381 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://centos.org/"
2021-11-26T17:27:02.382 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-11-26T17:27:02.382 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2021-11-26T17:27:02.382 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2021-11-26T17:27:02.383 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server
2021-11-26T17:27:02.483 INFO:teuthology.dispatcher.supervisor:Running job 6528387
2021-11-26T17:27:02.483 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_a0bc318848afe71da1fbf34e1e1b44738d8e3d6d/virtualenv/bin/teuthology -v --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2021-11-26_15:35:27-rados-wip-sage4-testing-2021-11-25-1626-distro-basic-smithi/6528387 --name sage-2021-11-26_15:35:27-rados-wip-sage4-testing-2021-11-25-1626-distro-basic-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/sage-2021-11-26_15:35:27-rados-wip-sage4-testing-2021-11-25-1626-distro-basic-smithi/6528387/orig.config.yaml
2021-11-26T17:27:02.487 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sage-2021-11-26_15:35:27-rados-wip-sage4-testing-2021-11-25-1626-distro-basic-smithi/6528387
2021-11-26T17:27:02.488 INFO:teuthology.dispatcher.supervisor:Job PID: 2618
2021-11-26T17:27:02.488 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-27T05:27:58.647 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-27T05:27:58.873 INFO:teuthology.kill:Killing Pids: {2618}
2021-11-27T05:28:00.036 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2021-11-27T05:28:00.037 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2021-11-27T05:28:00.037 INFO:teuthology.misc:Compressing logs...
2021-11-27T05:28:00.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com'
2021-11-27T05:28:00.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com'
2021-11-27T05:28:00.272 INFO:teuthology.orchestra.run.smithi137.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2021-11-27T05:28:00.456 INFO:teuthology.orchestra.run.smithi143.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2021-11-27T05:28:00.505 INFO:teuthology.orchestra.run.smithi137.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-11-27T05:28:00.506 INFO:teuthology.orchestra.run.smithi137.stderr:tar: Error is not recoverable: exiting now
2021-11-27T05:28:00.611 INFO:teuthology.orchestra.run.smithi143.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-11-27T05:28:00.612 INFO:teuthology.orchestra.run.smithi143.stderr:tar: Error is not recoverable: exiting now
2021-11-27T05:28:00.613 INFO:teuthology.misc:Compressing logs...
2021-11-27T05:28:00.974 INFO:teuthology.orchestra.run.smithi137.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-11-27T05:28:01.171 INFO:teuthology.misc:Compressing logs...
2021-11-27T05:28:02.354 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/6f27148e-4edf-11ec-8c2d-001a4aab830c/ceph-mgr.smithi143.tiukoy.log: file size changed while zipping
2021-11-27T05:28:04.456 INFO:teuthology.orchestra.run.smithi137.stderr:gzip: /var/log/ceph/6f27148e-4edf-11ec-8c2d-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2021-11-27T05:28:04.769 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/6f27148e-4edf-11ec-8c2d-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2021-11-27T05:28:06.253 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/6f27148e-4edf-11ec-8c2d-001a4aab830c/ceph-mon.smithi143.log: file size changed while zipping
2021-11-27T05:28:07.900 INFO:teuthology.orchestra.run.smithi137.stderr:gzip: /var/log/ceph/6f27148e-4edf-11ec-8c2d-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2021-11-27T05:28:08.843 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/6f27148e-4edf-11ec-8c2d-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2021-11-27T05:28:10.303 INFO:teuthology.orchestra.run.smithi137.stderr:gzip: /var/log/ceph/6f27148e-4edf-11ec-8c2d-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2021-11-27T05:28:11.772 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/6f27148e-4edf-11ec-8c2d-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2021-11-27T05:28:13.859 INFO:teuthology.orchestra.run.smithi137.stderr:gzip: /var/log/ceph/6f27148e-4edf-11ec-8c2d-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2021-11-27T05:28:14.766 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/6f27148e-4edf-11ec-8c2d-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2021-11-27T05:28:17.322 INFO:teuthology.orchestra.run.smithi137.stderr:gzip: /var/log/ceph/6f27148e-4edf-11ec-8c2d-001a4aab830c/ceph-mon.smithi137.log: file size changed while zipping
2021-11-27T05:29:05.838 INFO:teuthology.kill:No teuthology processes running
2021-11-27T05:29:07.016 INFO:teuthology.kill:Nuking machines: ['smithi137', 'smithi143']
2021-11-27T05:29:07.998 INFO:teuthology.kill:2021-11-27 05:29:07,997.997 INFO:teuthology.nuke:targets:
2021-11-27T05:29:07.999 INFO:teuthology.kill:  smithi137.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDO5tcYXtZRW8vJvZyaji7EGzw39GIN8H99rZdBF7UNXvhQt760qyQV6Gp57aJ3rj6CKTc1DknN8eKQ13QtK8xvOBZ3dMPMqNxOCBDTORSTxBfcktPTmKMmQOAdasq6W0zQ6KL4eacwYc5wgJEflnCe1k1Bl3fZm/S/kZTrJA9mXQIzFSSU0rmeSOsxhmRZmYDUEeOO6CSS0mNVTHu27nfAQnneKoJrBapY3Srj52emYdex1po6u56c59bZWjMEcis7Qex9HI7fmphJMLcdsHueGdtIwxkJ8vT7r8ufcRWY+4ACpY7HbxP4JESw6c2TvOy1WtHX6tn+OOSfgQKoVs7YxpE7rjcxO0gw9ahq695sAwWSfoqzhFqdamhVYh+Jsb9cWEU/+droiPtyNADsgajvWhRlDztL3wSZvM/o1TD2zAkwHGw9bG8NfXElQ+nEjgxCRFNsQ4u6AxbWBkHnVlm4OR4edJENjFg0j4l7N4RlX6H1kaVF0RSvIt6MTKr5HbU=
2021-11-27T05:29:07.999 INFO:teuthology.kill:  smithi143.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmZXG5rAKeR0tgskleWbg+GuNTX39n+ZG+Jw9lF2WJhhmL2rCLhiuH27qCnQAZeVlbxfb07dlHBhHgtzFfmTKwN5yCRxC+R+x+KG4FkekmjeRpHmdRouflt2N7xkjLOLM5Qem5MEvSyLbPeVEB7M5JT6YlwN6VOHkVOq0EJjDfwnVlJJMs04nDB2NAm0iHpbkn/mtltN8rahC4DW9tRqQ5c0FBj5+Y8A4pvElM3qlrc453N077pVLU5MOb+12YxGKCSwBgi1Fci5Oy+Vn89bRKRYuvbtMNnzii8RAcOt6DwxLnJSwUh+wN7vEOa5NJITa8Ih1lOxWL3mVwDx4Cwbrm7c7nfdWGJtTGLcwZFK8Et20dXU+LcKY5S2IKygaszyULI/i7xXuQLP6nljNBx56sNwSpeoQxGDR+seyIgSgZ8DB7hslEBQ8T7mi7CE08dSIHKVqcpApXLf8/rhUtsDIzmkDtGFcVZdzkY7BMP0YOC5hsn6VwkLBxY1IyesZjAwc=
2021-11-27T05:29:08.025 INFO:teuthology.kill:2021-11-27 05:29:08,025.025 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-27T05:29:08.032 INFO:teuthology.kill:2021-11-27 05:29:08,032.032 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-27T05:29:08.108 INFO:teuthology.kill:2021-11-27 05:29:08,107.107 INFO:teuthology.orchestra.console:Power off smithi143
2021-11-27T05:29:08.121 INFO:teuthology.kill:2021-11-27 05:29:08,121.121 INFO:teuthology.orchestra.console:Power off smithi137
2021-11-27T05:29:16.476 INFO:teuthology.kill:2021-11-27 05:29:16,476.476 INFO:teuthology.orchestra.console:Power off for smithi143 completed
2021-11-27T05:29:16.623 INFO:teuthology.kill:2021-11-27 05:29:16,622.622 INFO:teuthology.lock.ops:unlocked: smithi143.front.sepia.ceph.com
2021-11-27T05:29:20.592 INFO:teuthology.kill:2021-11-27 05:29:20,592.592 INFO:teuthology.orchestra.console:Power off for smithi137 completed
2021-11-27T05:29:20.739 INFO:teuthology.kill:2021-11-27 05:29:20,739.739 INFO:teuthology.lock.ops:unlocked: smithi137.front.sepia.ceph.com
2021-11-27T05:31:21.144 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-27T05:31:21.244 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi137 but it was locked by another job: /home/teuthworker/archive/akupczyk-2021-11-26_16:33:01-rados-pacific-bluefs-fine-grain-locking-4-distro-default-smithi/6529138
2021-11-27T05:31:21.244 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi143 but it was locked by another job: /home/teuthworker/archive/akupczyk-2021-11-26_16:33:01-rados-pacific-bluefs-fine-grain-locking-4-distro-default-smithi/6529136