2022-04-06T16:53:20.602 INFO:root:teuthology version: 1.1.0-3dd79763
2022-04-06T16:53:20.730 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi033.front.sepia.ceph.com' reimaging
2022-04-06T16:53:20.731 INFO:teuthology.lock.ops:Updating [ubuntu@smithi033.front.sepia.ceph.com]: reset os type and version on server
2022-04-06T16:53:20.732 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server
2022-04-06T16:53:20.754 INFO:teuthology.lock.ops:Node 'ubuntu@smithi033.front.sepia.ceph.com' reimaging is complete
2022-04-06T16:53:20.754 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi124.front.sepia.ceph.com' reimaging
2022-04-06T16:53:20.755 INFO:teuthology.lock.ops:Updating [ubuntu@smithi124.front.sepia.ceph.com]: reset os type and version on server
2022-04-06T16:53:20.755 INFO:teuthology.lock.ops:Updating smithi124.front.sepia.ceph.com on lock server
2022-04-06T16:53:20.778 INFO:teuthology.lock.ops:Node 'ubuntu@smithi124.front.sepia.ceph.com' reimaging is complete
2022-04-06T16:53:20.976 INFO:teuthology.provision.fog.smithi033:Scheduling deploy of centos 8.stream
2022-04-06T16:53:21.004 INFO:teuthology.provision.fog.smithi124:Scheduling deploy of centos 8.stream
2022-04-06T16:53:21.305 INFO:teuthology.orchestra.console:Power off smithi033
2022-04-06T16:53:21.344 INFO:teuthology.orchestra.console:Power off smithi124
2022-04-06T16:53:29.678 INFO:teuthology.orchestra.console:Power off for smithi033 completed
2022-04-06T16:53:29.779 INFO:teuthology.orchestra.console:Power on smithi033
2022-04-06T16:53:33.817 INFO:teuthology.orchestra.console:Power off for smithi124 completed
2022-04-06T16:53:33.919 INFO:teuthology.orchestra.console:Power on smithi124
2022-04-06T16:53:38.145 INFO:teuthology.orchestra.console:Power on for smithi033 completed
2022-04-06T16:53:38.246 INFO:teuthology.provision.fog.smithi033:Waiting for deploy to finish
2022-04-06T16:53:42.276 INFO:teuthology.orchestra.console:Power on for smithi124 completed
2022-04-06T16:53:42.377 INFO:teuthology.provision.fog.smithi124:Waiting for deploy to finish
2022-04-06T16:55:54.303 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed.
2022-04-06T16:57:53.324 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-04-06T16:57:53.638 INFO:teuthology.provision.fog.smithi033:Node is ready
2022-04-06T16:57:53.658 INFO:teuthology.orchestra.run.smithi033.stdout:smithi033.front.sepia.ceph.com
2022-04-06T16:57:53.718 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33 smithi033.front.sepia.ceph.com smithi033
2022-04-06T16:57:54.085 INFO:teuthology.provision.fog.smithi033:Deploy complete!
2022-04-06T16:58:20.447 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-04-06T16:58:20.748 INFO:teuthology.provision.fog.smithi124:Node is ready
2022-04-06T16:58:20.767 INFO:teuthology.orchestra.run.smithi124.stdout:smithi124.front.sepia.ceph.com
2022-04-06T16:58:20.827 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.15.124 smithi124.front.sepia.ceph.com smithi124
2022-04-06T16:58:21.187 INFO:teuthology.provision.fog.smithi124:Deploy complete!
2022-04-06T16:58:21.338 INFO:teuthology.lock.ops:Checking smithi033.front.sepia.ceph.com
2022-04-06T16:58:21.339 INFO:teuthology.lock.ops:New key found. Updating...
2022-04-06T16:58:21.361 INFO:teuthology.lock.ops:Checking smithi124.front.sepia.ceph.com
2022-04-06T16:58:21.361 INFO:teuthology.lock.ops:New key found. Updating...
2022-04-06T16:58:21.379 INFO:teuthology.lock.ops:Updating [smithi033.front.sepia.ceph.com]: set os type and version on server
2022-04-06T16:58:21.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com'
2022-04-06T16:58:21.579 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64
2022-04-06T16:58:21.639 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Stream"
2022-04-06T16:58:21.640 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="8"
2022-04-06T16:58:21.640 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos"
2022-04-06T16:58:21.640 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora"
2022-04-06T16:58:21.640 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="8"
2022-04-06T16:58:21.641 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el8"
2022-04-06T16:58:21.641 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Stream 8"
2022-04-06T16:58:21.641 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31"
2022-04-06T16:58:21.641 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-04-06T16:58:21.641 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://centos.org/"
2022-04-06T16:58:21.642 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-04-06T16:58:21.642 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-04-06T16:58:21.642 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-04-06T16:58:21.643 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server
2022-04-06T16:58:21.661 INFO:teuthology.lock.ops:Updating [smithi124.front.sepia.ceph.com]: set os type and version on server
2022-04-06T16:58:21.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com'
2022-04-06T16:58:21.858 INFO:teuthology.orchestra.run.smithi124.stdout:x86_64
2022-04-06T16:58:21.917 INFO:teuthology.orchestra.run.smithi124.stdout:NAME="CentOS Stream"
2022-04-06T16:58:21.917 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION="8"
2022-04-06T16:58:21.918 INFO:teuthology.orchestra.run.smithi124.stdout:ID="centos"
2022-04-06T16:58:21.918 INFO:teuthology.orchestra.run.smithi124.stdout:ID_LIKE="rhel fedora"
2022-04-06T16:58:21.918 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION_ID="8"
2022-04-06T16:58:21.918 INFO:teuthology.orchestra.run.smithi124.stdout:PLATFORM_ID="platform:el8"
2022-04-06T16:58:21.919 INFO:teuthology.orchestra.run.smithi124.stdout:PRETTY_NAME="CentOS Stream 8"
2022-04-06T16:58:21.919 INFO:teuthology.orchestra.run.smithi124.stdout:ANSI_COLOR="0;31"
2022-04-06T16:58:21.919 INFO:teuthology.orchestra.run.smithi124.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-04-06T16:58:21.919 INFO:teuthology.orchestra.run.smithi124.stdout:HOME_URL="https://centos.org/"
2022-04-06T16:58:21.919 INFO:teuthology.orchestra.run.smithi124.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-04-06T16:58:21.920 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-04-06T16:58:21.920 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-04-06T16:58:21.920 INFO:teuthology.lock.ops:Updating smithi124.front.sepia.ceph.com on lock server
2022-04-06T16:58:22.019 INFO:teuthology.dispatcher.supervisor:Running job 6779476
2022-04-06T16:58:22.020 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_3dd797633f37e2d7e2c4ae3ef9ac35a188c2028d/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-04-06_14:02:46-rados-wip-yuri4-testing-2022-04-05-1720-pacific-distro-default-smithi/6779476 --name yuriw-2022-04-06_14:02:46-rados-wip-yuri4-testing-2022-04-05-1720-pacific-distro-default-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2022-04-06_14:02:46-rados-wip-yuri4-testing-2022-04-05-1720-pacific-distro-default-smithi/6779476/orig.config.yaml
2022-04-06T16:58:22.023 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-04-06_14:02:46-rados-wip-yuri4-testing-2022-04-05-1720-pacific-distro-default-smithi/6779476
2022-04-06T16:58:22.023 INFO:teuthology.dispatcher.supervisor:Job PID: 24504
2022-04-06T16:58:22.023 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-04-06T23:28:32.986 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-04-06T23:28:33.160 INFO:teuthology.kill:Killing Pids: {24504}
2022-04-06T23:28:34.370 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-04-06T23:28:34.371 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-04-06T23:28:34.371 INFO:teuthology.misc:Compressing logs...
2022-04-06T23:28:34.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com'
2022-04-06T23:28:34.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com'
2022-04-06T23:28:34.617 INFO:teuthology.orchestra.run.smithi033.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-04-06T23:28:34.804 INFO:teuthology.orchestra.run.smithi124.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-04-06T23:28:34.857 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-04-06T23:28:34.857 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now
2022-04-06T23:28:34.900 INFO:teuthology.orchestra.run.smithi124.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-04-06T23:28:34.901 INFO:teuthology.orchestra.run.smithi124.stderr:tar: Error is not recoverable: exiting now
2022-04-06T23:28:34.902 INFO:teuthology.misc:Compressing logs...
2022-04-06T23:28:35.202 INFO:teuthology.misc:Compressing logs...
2022-04-06T23:28:51.701 INFO:teuthology.orchestra.run.smithi124.stderr:gzip: /var/log/ceph/6917f434-b5cb-11ec-8c36-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-04-06T23:28:53.737 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/6917f434-b5cb-11ec-8c36-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-04-06T23:29:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/6917f434-b5cb-11ec-8c36-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-04-06T23:29:08.210 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/6917f434-b5cb-11ec-8c36-001a4aab830c/ceph-mgr.smithi033.vmxilp.log: file size changed while zipping
2022-04-06T23:29:11.461 INFO:teuthology.orchestra.run.smithi124.stderr:gzip: /var/log/ceph/6917f434-b5cb-11ec-8c36-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-04-06T23:29:14.856 INFO:teuthology.orchestra.run.smithi124.stderr:gzip: /var/log/ceph/6917f434-b5cb-11ec-8c36-001a4aab830c/ceph-mon.smithi124.log: file size changed while zipping
2022-04-06T23:29:26.967 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/6917f434-b5cb-11ec-8c36-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-04-06T23:29:30.619 INFO:teuthology.orchestra.run.smithi124.stderr:gzip: /var/log/ceph/6917f434-b5cb-11ec-8c36-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-04-06T23:29:39.734 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/6917f434-b5cb-11ec-8c36-001a4aab830c/ceph-mon.smithi033.log: file size changed while zipping
2022-04-06T23:29:44.645 INFO:teuthology.orchestra.run.smithi124.stderr:gzip: /var/log/ceph/6917f434-b5cb-11ec-8c36-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-04-06T23:29:56.059 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/6917f434-b5cb-11ec-8c36-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-04-06T23:30:26.954 INFO:teuthology.kill:No teuthology processes running
2022-04-06T23:30:28.207 INFO:teuthology.kill:Nuking machines: ['smithi033', 'smithi124']
2022-04-06T23:30:29.168 INFO:teuthology.kill:2022-04-06 23:30:29,168.168 INFO:teuthology.nuke:targets:
2022-04-06T23:30:29.169 INFO:teuthology.kill:  smithi033.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1me4N9BRlUaeAy2mj/EM9SOm8GAwpiD3d/9AM17nhPwuQ9VOoFPWY1ncUg/xbTCrA6o28skh+uRq2xB8ffq2g4VLxgliJZs0jbf4uQ5/WddqeF3tBwrizph4KZk/X5RYzNLM6yUEgT0tIWsn+LBCn6qMsRxXXrCXQ45hgYdMBek4VwTvGsdcU698yHg1eE0a2F5MbJ+5nGAj+bn35iqSi8qz15WaJVczrpZ2xSWIZ4/7aY3MPh6EOGEUsSem6c4H6wvGWERpS0KCkN4x0xFxghWbBVXd0SGrKsz768HJDrSIYnQugt2bmNiEr/hQHyjuizJMk9ehQAvSZikAcxtFyT9AUm0qjIUMcPXXEKq7rjXcWZj9s0C81ClJYxb6hkJhA7wt8KbIdVK+hCxjIOtz15umTMb3jrHOZj7cMTMHjqDdGLiyFnhqs4T6oI6Py7MFxzioy64jDj60ohqfMA440OR0VF2AgXdPc4XXmuS6MCnrIbXCUhI2nteg2CGeg/OM=
2022-04-06T23:30:29.169 INFO:teuthology.kill:  smithi124.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnZaeyhWQbAbSgP0YMKYSUV0Z3m72Q7f6/mugZ5x3jP52fXmjrdeZgS5XfVd5tgPTBQMEOyH4ixBYHxkulQU25uRB7RxqaXKXhfZ3Lgji4SUGTtx6cacIHS2sbhbK9Fv8Mk1rlWP81Ued9xsgKyNKAukgvKVd/OYKFGiwaTGNd3jfR6/8ITm18wYIozfbizh7AMhtl8C/PF9Z4ThRKgM84uF0neyJ0SsADbcEEGgX6ZEAqJqrJILczdB/791TRa1uolFDu9kG+2UVCyOgXACi7NaDS60lGNHods/2tv1+gnSvf6kU3w7F01anpQyDsRfM8rCvHH51zEWst3KYuVrl80qRJoPLCOPkP/wetPZla0bEyb+gyBmKHn5TTQHu0G0GcvUfABpXZnkPHkPuseSpoRYJhAYac1XTni7whCzgS5rhIoHvnGQkGX3Yw+99pCmCJTy+BVMYyTn9tK6ISvkNyzcDc7v6JONbMARe0Ex4Se+4xngdQC/+IX7b1q6p90zU=
2022-04-06T23:30:29.194 INFO:teuthology.kill:2022-04-06 23:30:29,194.194 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-04-06T23:30:29.198 INFO:teuthology.kill:2022-04-06 23:30:29,198.198 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-04-06T23:30:29.284 INFO:teuthology.kill:2022-04-06 23:30:29,283.283 INFO:teuthology.orchestra.console:Power off smithi124
2022-04-06T23:30:29.297 INFO:teuthology.kill:2022-04-06 23:30:29,296.296 INFO:teuthology.orchestra.console:Power off smithi033
2022-04-06T23:30:37.663 INFO:teuthology.kill:2022-04-06 23:30:37,663.663 INFO:teuthology.orchestra.console:Power off for smithi124 completed
2022-04-06T23:30:37.670 INFO:teuthology.kill:2022-04-06 23:30:37,669.669 INFO:teuthology.orchestra.console:Power off for smithi033 completed
2022-04-06T23:30:37.816 INFO:teuthology.kill:2022-04-06 23:30:37,816.816 INFO:teuthology.lock.ops:unlocked: smithi124.front.sepia.ceph.com
2022-04-06T23:30:37.819 INFO:teuthology.kill:2022-04-06 23:30:37,819.819 INFO:teuthology.lock.ops:unlocked: smithi033.front.sepia.ceph.com
2022-04-06T23:32:38.043 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-04-06T23:32:38.137 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi033 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-04-06_14:21:25-orch-wip-yuri2-testing-2022-03-31-1523-pacific-distro-default-smithi/6779730
2022-04-06T23:32:38.137 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi124 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-04-06_14:21:25-orch-wip-yuri2-testing-2022-03-31-1523-pacific-distro-default-smithi/6779730