2021-09-18T01:55:48.554 INFO:root:teuthology version: 1.1.0-4e1f59de 2021-09-18T01:55:48.693 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging 2021-09-18T01:55:48.694 INFO:teuthology.lock.ops:Updating [ubuntu@smithi142.front.sepia.ceph.com]: reset os type and version on server 2021-09-18T01:55:48.694 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2021-09-18T01:55:48.713 INFO:teuthology.lock.ops:Node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging is complete 2021-09-18T01:55:48.714 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi154.front.sepia.ceph.com' reimaging 2021-09-18T01:55:48.714 INFO:teuthology.lock.ops:Updating [ubuntu@smithi154.front.sepia.ceph.com]: reset os type and version on server 2021-09-18T01:55:48.714 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2021-09-18T01:55:48.729 INFO:teuthology.lock.ops:Node 'ubuntu@smithi154.front.sepia.ceph.com' reimaging is complete 2021-09-18T01:55:48.730 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi156.front.sepia.ceph.com' reimaging 2021-09-18T01:55:48.730 INFO:teuthology.lock.ops:Updating [ubuntu@smithi156.front.sepia.ceph.com]: reset os type and version on server 2021-09-18T01:55:48.730 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2021-09-18T01:55:48.745 INFO:teuthology.lock.ops:Node 'ubuntu@smithi156.front.sepia.ceph.com' reimaging is complete 2021-09-18T01:55:48.922 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of ubuntu 20.04 2021-09-18T01:55:48.959 INFO:teuthology.provision.fog.smithi154:Scheduling deploy of ubuntu 20.04 2021-09-18T01:55:48.968 INFO:teuthology.provision.fog.smithi156:Scheduling deploy of ubuntu 20.04 2021-09-18T01:55:49.272 INFO:teuthology.orchestra.console:Power off smithi142 2021-09-18T01:55:49.315 INFO:teuthology.orchestra.console:Power off smithi154 2021-09-18T01:55:49.393 INFO:teuthology.orchestra.console:Power off smithi156 2021-09-18T01:55:57.667 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2021-09-18T01:55:57.768 INFO:teuthology.orchestra.console:Power on smithi142 2021-09-18T01:55:57.776 INFO:teuthology.orchestra.console:Power off for smithi156 completed 2021-09-18T01:55:57.878 INFO:teuthology.orchestra.console:Power on smithi156 2021-09-18T01:56:01.800 INFO:teuthology.orchestra.console:Power off for smithi154 completed 2021-09-18T01:56:01.902 INFO:teuthology.orchestra.console:Power on smithi154 2021-09-18T01:56:06.138 INFO:teuthology.orchestra.console:Power on for smithi142 completed 2021-09-18T01:56:06.156 INFO:teuthology.orchestra.console:Power on for smithi154 completed 2021-09-18T01:56:06.239 INFO:teuthology.orchestra.console:Power on for smithi156 completed 2021-09-18T01:56:06.239 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish 2021-09-18T01:56:06.257 INFO:teuthology.provision.fog.smithi154:Waiting for deploy to finish 2021-09-18T01:56:06.340 INFO:teuthology.provision.fog.smithi156:Waiting for deploy to finish 2021-09-18T02:01:03.682 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-18T02:01:04.154 INFO:teuthology.provision.fog.smithi142:Node is ready 2021-09-18T02:01:04.161 INFO:teuthology.orchestra.run.smithi142.stdout:smithi142.front.sepia.ceph.com 2021-09-18T02:01:04.170 INFO:teuthology.orchestra.run.smithi142.stdout:172.21.15.142 smithi142.front.sepia.ceph.com smithi142 2021-09-18T02:01:04.412 INFO:teuthology.provision.fog.smithi142:Deploy complete! 2021-09-18T02:01:18.904 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-18T02:01:19.400 INFO:teuthology.provision.fog.smithi156:Node is ready 2021-09-18T02:01:19.406 INFO:teuthology.orchestra.run.smithi156.stdout:smithi156.front.sepia.ceph.com 2021-09-18T02:01:19.454 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.15.156 smithi156.front.sepia.ceph.com smithi156 2021-09-18T02:01:19.652 INFO:teuthology.provision.fog.smithi156:Deploy complete! 2021-09-18T02:01:23.209 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-18T02:01:23.694 INFO:teuthology.provision.fog.smithi154:Node is ready 2021-09-18T02:01:23.743 INFO:teuthology.orchestra.run.smithi154.stdout:smithi154.front.sepia.ceph.com 2021-09-18T02:01:23.751 INFO:teuthology.orchestra.run.smithi154.stdout:172.21.15.154 smithi154.front.sepia.ceph.com smithi154 2021-09-18T02:01:23.957 INFO:teuthology.provision.fog.smithi154:Deploy complete! 2021-09-18T02:01:24.167 INFO:teuthology.lock.ops:Checking smithi142.front.sepia.ceph.com 2021-09-18T02:01:24.168 INFO:teuthology.lock.ops:New key found. Updating... 2021-09-18T02:01:24.187 INFO:teuthology.lock.ops:Checking smithi154.front.sepia.ceph.com 2021-09-18T02:01:24.187 INFO:teuthology.lock.ops:New key found. Updating... 2021-09-18T02:01:24.203 INFO:teuthology.lock.ops:Checking smithi156.front.sepia.ceph.com 2021-09-18T02:01:24.204 INFO:teuthology.lock.ops:New key found. Updating... 2021-09-18T02:01:24.219 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server 2021-09-18T02:01:24.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2021-09-18T02:01:24.483 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2021-09-18T02:01:24.487 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Ubuntu" 2021-09-18T02:01:24.488 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-09-18T02:01:24.488 INFO:teuthology.orchestra.run.smithi142.stdout:ID=ubuntu 2021-09-18T02:01:24.488 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE=debian 2021-09-18T02:01:24.488 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-09-18T02:01:24.489 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="20.04" 2021-09-18T02:01:24.489 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.ubuntu.com/" 2021-09-18T02:01:24.489 INFO:teuthology.orchestra.run.smithi142.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-09-18T02:01:24.489 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-09-18T02:01:24.489 INFO:teuthology.orchestra.run.smithi142.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-09-18T02:01:24.490 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_CODENAME=focal 2021-09-18T02:01:24.490 INFO:teuthology.orchestra.run.smithi142.stdout:UBUNTU_CODENAME=focal 2021-09-18T02:01:24.491 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2021-09-18T02:01:24.507 INFO:teuthology.lock.ops:Updating [smithi154.front.sepia.ceph.com]: set os type and version on server 2021-09-18T02:01:24.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2021-09-18T02:01:24.711 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2021-09-18T02:01:24.717 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="Ubuntu" 2021-09-18T02:01:24.717 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-09-18T02:01:24.718 INFO:teuthology.orchestra.run.smithi154.stdout:ID=ubuntu 2021-09-18T02:01:24.718 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE=debian 2021-09-18T02:01:24.718 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-09-18T02:01:24.718 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="20.04" 2021-09-18T02:01:24.719 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://www.ubuntu.com/" 2021-09-18T02:01:24.719 INFO:teuthology.orchestra.run.smithi154.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-09-18T02:01:24.719 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-09-18T02:01:24.719 INFO:teuthology.orchestra.run.smithi154.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-09-18T02:01:24.719 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_CODENAME=focal 2021-09-18T02:01:24.720 INFO:teuthology.orchestra.run.smithi154.stdout:UBUNTU_CODENAME=focal 2021-09-18T02:01:24.720 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2021-09-18T02:01:24.736 INFO:teuthology.lock.ops:Updating [smithi156.front.sepia.ceph.com]: set os type and version on server 2021-09-18T02:01:24.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2021-09-18T02:01:24.953 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2021-09-18T02:01:24.960 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="Ubuntu" 2021-09-18T02:01:24.960 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-09-18T02:01:24.960 INFO:teuthology.orchestra.run.smithi156.stdout:ID=ubuntu 2021-09-18T02:01:24.960 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE=debian 2021-09-18T02:01:24.961 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-09-18T02:01:24.961 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="20.04" 2021-09-18T02:01:24.961 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://www.ubuntu.com/" 2021-09-18T02:01:24.961 INFO:teuthology.orchestra.run.smithi156.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-09-18T02:01:24.962 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-09-18T02:01:24.962 INFO:teuthology.orchestra.run.smithi156.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-09-18T02:01:24.962 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_CODENAME=focal 2021-09-18T02:01:24.962 INFO:teuthology.orchestra.run.smithi156.stdout:UBUNTU_CODENAME=focal 2021-09-18T02:01:24.963 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2021-09-18T02:01:25.059 INFO:teuthology.dispatcher.supervisor:Running job 6395426 2021-09-18T02:01:25.059 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_4e1f59de82924f116bd6b91664f90ef632c60a3f/virtualenv/bin/teuthology -v --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2021-09-17_20:49:46-fs-wip-pdonnell-testing-20210917.174826-distro-basic-smithi/6395426 --name pdonnell-2021-09-17_20:49:46-fs-wip-pdonnell-testing-20210917.174826-distro-basic-smithi --description fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} -- /home/teuthworker/archive/pdonnell-2021-09-17_20:49:46-fs-wip-pdonnell-testing-20210917.174826-distro-basic-smithi/6395426/orig.config.yaml 2021-09-18T02:01:25.063 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/pdonnell-2021-09-17_20:49:46-fs-wip-pdonnell-testing-20210917.174826-distro-basic-smithi/6395426 2021-09-18T02:01:25.064 INFO:teuthology.dispatcher.supervisor:Job PID: 9471 2021-09-18T02:01:25.064 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-09-18T14:01:40.688 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-09-18T14:01:40.925 INFO:teuthology.kill:Killing Pids: {9471} 2021-09-18T14:01:41.943 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2021-09-18T14:01:41.944 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2021-09-18T14:01:41.944 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2021-09-18T14:01:41.944 INFO:teuthology.misc:Compressing logs... 2021-09-18T14:01:41.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2021-09-18T14:01:42.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2021-09-18T14:01:42.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2021-09-18T14:01:42.646 INFO:teuthology.misc:Compressing logs... 2021-09-18T14:01:42.841 INFO:teuthology.misc:Compressing logs... 2021-09-18T14:07:56.750 INFO:teuthology.kill:No teuthology processes running 2021-09-18T14:07:57.749 INFO:teuthology.kill:Nuking machines: ['smithi142', 'smithi154', 'smithi156'] 2021-09-18T14:07:58.572 INFO:teuthology.kill:2021-09-18 14:07:58,572.572 INFO:teuthology.nuke:targets: 2021-09-18T14:07:58.573 INFO:teuthology.kill: smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPHzSR2kmXs+Lyg/ZBSyCsRl+DOcI1vTe7RGosfGX/iob1zsYdwEQsd/TfZyP06UCX5DTJXlbRZAqIowR/+9eurS6nzk8kAO3JDI6Sv3xk6Wd+Lyf+Lohl6aUWfzwn9ic1gsl0C57GwXHptGeAj16+derBpEvL2chrXt8oqTp+naBFj8/It9D8JiUpHV3/PpE3NWT6n+tUAFi1OtRUhdMpLYiYGauH9X6i/EHUNqsyltds8PATpVDpxkkRMAns8feNc5ah0yc5W1EBZaSWoOwmq52byGTPaTnBYYOIfllvYl1mlMTxd4xpZbSgXOvurflxk82ikZraWnXglLZJ4r9PGvwwBMa7Q1MmRFRZNxzp2K9gagzR0bLJtu5NJfDIJ7IYiY9DF3JrslgyDiAEp0uWaQrlsdJF67mTkacZVQDhT1bVEbpOKHkKi+X1HYXcaIYeFbUaMKJkXIV3mVo9AQmYR/w3DooFQfpLZQkPmPF2iMeEem57s58IZY1yn2Z3DXE= 2021-09-18T14:07:58.573 INFO:teuthology.kill: smithi154.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCduzIjQkzcfOqvGbEXQRRsOfF4MqhAL4DVcyxfNFgOsXctFdIS43XM32AiS7bUStrdhIaTatPpu9znUJiYZPbBUGIeFSKD9ldsvaQb7j3ZuzqhC93fktv9fiCNzHCkLSVzrUJTDEYOTNekUgiVb+k6qVmq4CeczYFwNnIq9pCpydBSLa05wBtQKFx4+Z/RCtOB6PZpvWajCzhehJ18r/z0T70pZd1BKLvVllSk2wVCszZCEClzqwVbsWinomN7aw/I9jWjwTTLBuMTYRduCSgaYF2GojTog6cIOabkW9AG/kLuu2vl01evDb/iuYUEbLLgKH8zo1HE5X52sZb6B/yqzE0UloYMZ4I01ZeYB/bEJqDvne8pMZ+i15gwya2xsIO7InD4zTR5CK8c8enc+SNPzXbLlWUr9TKJqv6dAiRjPL/O3Jj0aEdddukx8kplNztyeD5ElzYkj/y5WxVKq1AJuSwbSSKtwCbKhUIPG9CnjrhnH0UStL/23QOXaBmHJg0= 2021-09-18T14:07:58.573 INFO:teuthology.kill: smithi156.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOnaN30adbB+Ir54eDLlky97rKGK1C7PWokniaovZe6nvusRR/6yTbZOfV6DpkMF7/Ks4o8ITgPlAI9CW6YyHcCJ7mRB3uFhhqT8z39L0cXKI8G59ZtjVbkpn9KlNxjOGbV5YbHk/T7dtC0Qtc+MMEuYcePE/CGd15gT17g0x7hlGNPg+DJfJvroBXwAUlBBBussk19+kP1z/9gIV9+Dgi7MEZ2K55e89WKx9hkFhsY3//oIIen07ho5ee19IsaCWeG1jnyf6CgfnY5fQePWHYY0K1k3rdmLD+y7T0aAy3R3bRylLughrJiRsaVQV+OObxULcDMVFvUbrjaAYd3ARYOmrTpHl5KGX8TXwWoibTg1HAwBrquiP86k7DuR+kv66jw7IFfMa4aZir8HwFfLLDHm1dj0idb7Nnrslr78CVw15EhXG7FdOwQ1dQaqJ6j8pCNGJkmGID45wjinjoF2vgEWtSXjpBv3rtJmn1gRddT+QQ5FFfUeXr3lm4jubDD/s= 2021-09-18T14:07:58.594 INFO:teuthology.kill:2021-09-18 14:07:58,594.594 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-18T14:07:58.598 INFO:teuthology.kill:2021-09-18 14:07:58,598.598 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-18T14:07:58.599 INFO:teuthology.kill:2021-09-18 14:07:58,599.599 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-18T14:07:58.655 INFO:teuthology.kill:2021-09-18 14:07:58,654.654 INFO:teuthology.orchestra.console:Power off smithi156 2021-09-18T14:07:58.669 INFO:teuthology.kill:2021-09-18 14:07:58,669.669 INFO:teuthology.orchestra.console:Power off smithi154 2021-09-18T14:07:58.679 INFO:teuthology.kill:2021-09-18 14:07:58,679.679 INFO:teuthology.orchestra.console:Power off smithi142 2021-09-18T14:08:07.041 INFO:teuthology.kill:2021-09-18 14:08:07,041.041 INFO:teuthology.orchestra.console:Power off for smithi154 completed 2021-09-18T14:08:07.198 INFO:teuthology.kill:2021-09-18 14:08:07,197.197 INFO:teuthology.lock.ops:unlocked smithi154.front.sepia.ceph.com 2021-09-18T14:08:11.147 INFO:teuthology.kill:2021-09-18 14:08:11,147.147 INFO:teuthology.orchestra.console:Power off for smithi156 completed 2021-09-18T14:08:11.153 INFO:teuthology.kill:2021-09-18 14:08:11,153.153 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2021-09-18T14:08:11.289 INFO:teuthology.kill:2021-09-18 14:08:11,289.289 INFO:teuthology.lock.ops:unlocked smithi156.front.sepia.ceph.com 2021-09-18T14:08:11.293 INFO:teuthology.kill:2021-09-18 14:08:11,293.293 INFO:teuthology.lock.ops:unlocked smithi142.front.sepia.ceph.com 2021-09-18T14:10:11.561 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15