2021-12-09T10:11:30.126 INFO:root:teuthology version: 1.1.0-95a7d479
2021-12-09T10:11:30.252 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi072.front.sepia.ceph.com' reimaging
2021-12-09T10:11:30.254 INFO:teuthology.lock.ops:Updating [ubuntu@smithi072.front.sepia.ceph.com]: reset os type and version on server
2021-12-09T10:11:30.254 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server
2021-12-09T10:11:30.274 INFO:teuthology.lock.ops:Node 'ubuntu@smithi072.front.sepia.ceph.com' reimaging is complete
2021-12-09T10:11:30.274 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi116.front.sepia.ceph.com' reimaging
2021-12-09T10:11:30.275 INFO:teuthology.lock.ops:Updating [ubuntu@smithi116.front.sepia.ceph.com]: reset os type and version on server
2021-12-09T10:11:30.275 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server
2021-12-09T10:11:30.294 INFO:teuthology.lock.ops:Node 'ubuntu@smithi116.front.sepia.ceph.com' reimaging is complete
2021-12-09T10:11:30.490 INFO:teuthology.provision.fog.smithi072:Scheduling deploy of centos 8.stream
2021-12-09T10:11:30.495 INFO:teuthology.provision.fog.smithi116:Scheduling deploy of centos 8.stream
2021-12-09T10:11:30.854 INFO:teuthology.orchestra.console:Power off smithi116
2021-12-09T10:11:30.870 INFO:teuthology.orchestra.console:Power off smithi072
2021-12-09T10:11:39.229 INFO:teuthology.orchestra.console:Power off for smithi116 completed
2021-12-09T10:11:39.330 INFO:teuthology.orchestra.console:Power on smithi116
2021-12-09T10:11:43.344 INFO:teuthology.orchestra.console:Power off for smithi072 completed
2021-12-09T10:11:43.446 INFO:teuthology.orchestra.console:Power on smithi072
2021-12-09T10:11:47.701 INFO:teuthology.orchestra.console:Power on for smithi072 completed
2021-12-09T10:11:47.803 INFO:teuthology.provision.fog.smithi072:Waiting for deploy to finish
2021-12-09T10:11:47.815 INFO:teuthology.orchestra.console:Power on for smithi116 completed
2021-12-09T10:11:47.917 INFO:teuthology.provision.fog.smithi116:Waiting for deploy to finish
2021-12-09T10:16:37.650 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-12-09T10:16:37.925 INFO:teuthology.provision.fog.smithi116:Node is ready
2021-12-09T10:16:37.942 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116.front.sepia.ceph.com
2021-12-09T10:16:37.999 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116 smithi116.front.sepia.ceph.com smithi116
2021-12-09T10:16:38.345 INFO:teuthology.provision.fog.smithi116:Deploy complete!
2021-12-09T10:17:07.703 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-12-09T10:17:08.061 INFO:teuthology.provision.fog.smithi072:Node is ready
2021-12-09T10:17:08.078 INFO:teuthology.orchestra.run.smithi072.stdout:smithi072.front.sepia.ceph.com
2021-12-09T10:17:08.135 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72 smithi072.front.sepia.ceph.com smithi072
2021-12-09T10:17:08.477 INFO:teuthology.provision.fog.smithi072:Deploy complete!
2021-12-09T10:17:08.630 INFO:teuthology.lock.ops:Checking smithi072.front.sepia.ceph.com
2021-12-09T10:17:08.630 INFO:teuthology.lock.ops:New key found. Updating...
2021-12-09T10:17:08.650 INFO:teuthology.lock.ops:Checking smithi116.front.sepia.ceph.com
2021-12-09T10:17:08.650 INFO:teuthology.lock.ops:New key found. Updating...
2021-12-09T10:17:08.668 INFO:teuthology.lock.ops:Updating [smithi072.front.sepia.ceph.com]: set os type and version on server
2021-12-09T10:17:08.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com'
2021-12-09T10:17:08.858 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64
2021-12-09T10:17:08.914 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Stream"
2021-12-09T10:17:08.914 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="8"
2021-12-09T10:17:08.915 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos"
2021-12-09T10:17:08.915 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora"
2021-12-09T10:17:08.915 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="8"
2021-12-09T10:17:08.915 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el8"
2021-12-09T10:17:08.915 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Stream 8"
2021-12-09T10:17:08.916 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31"
2021-12-09T10:17:08.916 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-12-09T10:17:08.916 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/"
2021-12-09T10:17:08.916 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-12-09T10:17:08.917 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2021-12-09T10:17:08.917 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2021-12-09T10:17:08.919 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server
2021-12-09T10:17:08.936 INFO:teuthology.lock.ops:Updating [smithi116.front.sepia.ceph.com]: set os type and version on server
2021-12-09T10:17:08.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com'
2021-12-09T10:17:09.179 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64
2021-12-09T10:17:09.235 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream"
2021-12-09T10:17:09.236 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="8"
2021-12-09T10:17:09.236 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos"
2021-12-09T10:17:09.236 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora"
2021-12-09T10:17:09.237 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="8"
2021-12-09T10:17:09.237 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el8"
2021-12-09T10:17:09.237 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 8"
2021-12-09T10:17:09.237 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31"
2021-12-09T10:17:09.238 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-12-09T10:17:09.238 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/"
2021-12-09T10:17:09.238 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-12-09T10:17:09.238 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2021-12-09T10:17:09.239 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2021-12-09T10:17:09.239 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server
2021-12-09T10:17:09.344 INFO:teuthology.dispatcher.supervisor:Running job 6553599
2021-12-09T10:17:09.344 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-12-08_20:57:58-rados-WIP_GBH_safe_fast_shutdown_6-distro-basic-smithi/6553599 --name benhanokh-2021-12-08_20:57:58-rados-WIP_GBH_safe_fast_shutdown_6-distro-basic-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/benhanokh-2021-12-08_20:57:58-rados-WIP_GBH_safe_fast_shutdown_6-distro-basic-smithi/6553599/orig.config.yaml
2021-12-09T10:17:09.348 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-12-08_20:57:58-rados-WIP_GBH_safe_fast_shutdown_6-distro-basic-smithi/6553599
2021-12-09T10:17:09.348 INFO:teuthology.dispatcher.supervisor:Job PID: 11240
2021-12-09T10:17:09.349 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-12-09T22:17:29.471 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-12-09T22:17:29.687 INFO:teuthology.kill:Killing Pids: {11240}
2021-12-09T22:17:31.060 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2021-12-09T22:17:31.062 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2021-12-09T22:17:31.063 INFO:teuthology.misc:Compressing logs...
2021-12-09T22:17:31.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com'
2021-12-09T22:17:31.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com'
2021-12-09T22:17:31.306 INFO:teuthology.orchestra.run.smithi072.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2021-12-09T22:17:31.499 INFO:teuthology.orchestra.run.smithi116.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2021-12-09T22:17:31.540 INFO:teuthology.orchestra.run.smithi072.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-12-09T22:17:31.541 INFO:teuthology.orchestra.run.smithi072.stderr:tar: Error is not recoverable: exiting now
2021-12-09T22:17:31.596 INFO:teuthology.orchestra.run.smithi116.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-12-09T22:17:31.597 INFO:teuthology.orchestra.run.smithi116.stderr:tar: Error is not recoverable: exiting now
2021-12-09T22:17:31.598 INFO:teuthology.misc:Compressing logs...
2021-12-09T22:17:31.959 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-12-09T22:17:32.157 INFO:teuthology.misc:Compressing logs...
2021-12-09T22:17:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:gzip: /var/log/ceph/b31e520c-58d9-11ec-8c30-001a4aab830c/ceph-mgr.smithi116.jesndu.log: file size changed while zipping
2021-12-09T22:17:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:gzip: /var/log/ceph/b31e520c-58d9-11ec-8c30-001a4aab830c/ceph-mon.smithi116.log: file size changed while zipping
2021-12-09T22:17:59.245 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/b31e520c-58d9-11ec-8c30-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2021-12-09T22:18:19.372 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/b31e520c-58d9-11ec-8c30-001a4aab830c/ceph-mon.smithi072.log: file size changed while zipping
2021-12-09T22:18:19.977 INFO:teuthology.orchestra.run.smithi116.stderr:gzip: /var/log/ceph/b31e520c-58d9-11ec-8c30-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2021-12-09T22:18:44.412 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/b31e520c-58d9-11ec-8c30-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2021-12-09T22:18:44.762 INFO:teuthology.orchestra.run.smithi116.stderr:gzip: /var/log/ceph/b31e520c-58d9-11ec-8c30-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2021-12-09T22:19:01.642 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/b31e520c-58d9-11ec-8c30-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2021-12-09T22:19:09.789 INFO:teuthology.orchestra.run.smithi116.stderr:gzip: /var/log/ceph/b31e520c-58d9-11ec-8c30-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2021-12-09T22:19:28.153 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/b31e520c-58d9-11ec-8c30-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2021-12-09T22:19:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:gzip: /var/log/ceph/b31e520c-58d9-11ec-8c30-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2021-12-09T22:20:22.602 INFO:teuthology.kill:No teuthology processes running
2021-12-09T22:20:23.902 INFO:teuthology.kill:Nuking machines: ['smithi072', 'smithi116']
2021-12-09T22:20:24.913 INFO:teuthology.kill:2021-12-09 22:20:24,913.913 INFO:teuthology.nuke:targets:
2021-12-09T22:20:24.915 INFO:teuthology.kill:  smithi072.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRvU4yFqBVlEPk7zX+4ezMmNIUdaWCJ9rcbM86whi02/yomRUU+L6BWa9I+I3jD40c1AUh6cm9M225mXMbg6nrqdqWH8VEx3O0ONcKoKM918mkcMb7Q1Rvon4EWoadGuT45kTwNzLE4M15Jc5IVpYrCF4TV8ui5Tk5Fwvnpd4UHbs/cEXewgoIGCwOythwCl8HzICTER81rULihwXz0luK/GQZZd/6iT8WRYJQ7vlMqXBqKCzskNmuYNFiGw8ubnG6OE1r+mGuJX/YtmW2GlH4u2qa7NfHMWuXPbQtNkZyfGNaJla+nQ67pze6hamhsglYmzs7DJB3PLLGmJ9vPURunxEX7X4S7buiYwH2iDXV7uIH1o8PAIWpEELGRpgS8B9wdwVucb9d4qZq4XH86TFDh/IBiOG5AF6jhCHKrDBGNByPuNbGs+iIzczQxv1S/0/1qIDWUvVbQSuxXQFKgHgM3WnEWeD5bzl/5kD2ZkTWxgb/BUJxAH76UEHtl9rZHx8=
2021-12-09T22:20:24.915 INFO:teuthology.kill:  smithi116.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwLawYNMpKs4dkcenTpYMsUQrq8DT19O2wZ9OjEufUmbXTBxYrL7Mgw2CkmVzNPMSX9WhguScDl94E8oYKlNujB3s9i7E8Hp+yQFmuHbcgw9jomgWooiG6K8QZZsRLU9emqTgcBQu5iGdTizcFyPrTYzp9LDoYBIwao91wbhdXWI/DNpuuQ8HiWwjJjuSb8aqYQznSTBQVupVYy5D5h7zerKpV5Y1kZkAROoy/ScQAHR8HZppw9mMK16/ng1eu+bpdrFaeEhm4N0h4ow82VGqo8DYk2rc4VoD+jupMMQorkD/PVb6i+p3AffWv3WXZ9GC2z5Q8I6lxgHPCHr+zW5lzE/LXQMpAaUzLgrdTwURfdjAl/WLHDAy15aEZbnTo6aaHOOxh/Iieo7JF4xGQr7q44HWjDhLYM6/mpyKbzWCugbRVE+TBSZJzF6IkbOYL03oMf/AXXA7icVCqPU3uMLgH/RQ7J1q6uqMy3N+QDixw7u8xa2oErKOvAInKaPlAbvs=
2021-12-09T22:20:24.944 INFO:teuthology.kill:2021-12-09 22:20:24,944.944 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-12-09T22:20:24.949 INFO:teuthology.kill:2021-12-09 22:20:24,949.949 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-12-09T22:20:25.027 INFO:teuthology.kill:2021-12-09 22:20:25,027.027 INFO:teuthology.orchestra.console:Power off smithi072
2021-12-09T22:20:25.040 INFO:teuthology.kill:2021-12-09 22:20:25,040.040 INFO:teuthology.orchestra.console:Power off smithi116
2021-12-09T22:20:33.409 INFO:teuthology.kill:2021-12-09 22:20:33,408.408 INFO:teuthology.orchestra.console:Power off for smithi116 completed
2021-12-09T22:20:33.565 INFO:teuthology.kill:2021-12-09 22:20:33,564.564 INFO:teuthology.lock.ops:unlocked: smithi116.front.sepia.ceph.com
2021-12-09T22:20:37.513 INFO:teuthology.kill:2021-12-09 22:20:37,512.512 INFO:teuthology.orchestra.console:Power off for smithi072 completed
2021-12-09T22:20:37.659 INFO:teuthology.kill:2021-12-09 22:20:37,659.659 INFO:teuthology.lock.ops:unlocked: smithi072.front.sepia.ceph.com
2021-12-09T22:22:38.083 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-12-09T22:22:38.239 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi072 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-12-09_21:14:34-rgw-wip-yuri10-testing-2021-12-08-0827-octopus-distro-default-smithi/6555814
2021-12-09T22:22:38.240 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi116 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-12-09_21:14:34-rgw-wip-yuri10-testing-2021-12-08-0827-octopus-distro-default-smithi/6555812