2021-12-12T19:10:03.567 INFO:root:teuthology version: 1.1.0-95a7d479
2021-12-12T19:10:03.735 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi050.front.sepia.ceph.com' reimaging
2021-12-12T19:10:03.737 INFO:teuthology.lock.ops:Updating [ubuntu@smithi050.front.sepia.ceph.com]: reset os type and version on server
2021-12-12T19:10:03.737 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server
2021-12-12T19:10:03.756 INFO:teuthology.lock.ops:Node 'ubuntu@smithi050.front.sepia.ceph.com' reimaging is complete
2021-12-12T19:10:03.757 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi055.front.sepia.ceph.com' reimaging
2021-12-12T19:10:03.757 INFO:teuthology.lock.ops:Updating [ubuntu@smithi055.front.sepia.ceph.com]: reset os type and version on server
2021-12-12T19:10:03.758 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server
2021-12-12T19:10:03.777 INFO:teuthology.lock.ops:Node 'ubuntu@smithi055.front.sepia.ceph.com' reimaging is complete
2021-12-12T19:10:03.778 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi090.front.sepia.ceph.com' reimaging
2021-12-12T19:10:03.778 INFO:teuthology.lock.ops:Updating [ubuntu@smithi090.front.sepia.ceph.com]: reset os type and version on server
2021-12-12T19:10:03.779 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server
2021-12-12T19:10:03.798 INFO:teuthology.lock.ops:Node 'ubuntu@smithi090.front.sepia.ceph.com' reimaging is complete
2021-12-12T19:10:03.968 INFO:teuthology.provision.fog.smithi050:Scheduling deploy of centos 8.3
2021-12-12T19:10:03.996 INFO:teuthology.provision.fog.smithi055:Scheduling deploy of centos 8.3
2021-12-12T19:10:04.012 INFO:teuthology.provision.fog.smithi090:Scheduling deploy of centos 8.3
2021-12-12T19:10:04.377 INFO:teuthology.orchestra.console:Power off smithi050
2021-12-12T19:10:04.391 INFO:teuthology.orchestra.console:Power off smithi090
2021-12-12T19:10:04.441 INFO:teuthology.orchestra.console:Power off smithi055
2021-12-12T19:10:12.760 INFO:teuthology.orchestra.console:Power off for smithi050 completed
2021-12-12T19:10:12.861 INFO:teuthology.orchestra.console:Power on smithi050
2021-12-12T19:10:16.881 INFO:teuthology.orchestra.console:Power off for smithi090 completed
2021-12-12T19:10:16.915 INFO:teuthology.orchestra.console:Power off for smithi055 completed
2021-12-12T19:10:16.982 INFO:teuthology.orchestra.console:Power on smithi090
2021-12-12T19:10:17.017 INFO:teuthology.orchestra.console:Power on smithi055
2021-12-12T19:10:21.229 INFO:teuthology.orchestra.console:Power on for smithi050 completed
2021-12-12T19:10:21.234 INFO:teuthology.orchestra.console:Power on for smithi090 completed
2021-12-12T19:10:21.287 INFO:teuthology.orchestra.console:Power on for smithi055 completed
2021-12-12T19:10:21.330 INFO:teuthology.provision.fog.smithi050:Waiting for deploy to finish
2021-12-12T19:10:21.335 INFO:teuthology.provision.fog.smithi090:Waiting for deploy to finish
2021-12-12T19:10:21.388 INFO:teuthology.provision.fog.smithi055:Waiting for deploy to finish
2021-12-12T19:14:56.961 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-12-12T19:14:57.286 INFO:teuthology.provision.fog.smithi050:Node is ready
2021-12-12T19:14:57.302 INFO:teuthology.orchestra.run.smithi050.stdout:smithi050.front.sepia.ceph.com
2021-12-12T19:14:57.359 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50 smithi050.front.sepia.ceph.com smithi050
2021-12-12T19:14:57.689 INFO:teuthology.provision.fog.smithi050:Deploy complete!
2021-12-12T19:15:28.291 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-12-12T19:15:28.608 INFO:teuthology.provision.fog.smithi055:Node is ready
2021-12-12T19:15:28.626 INFO:teuthology.orchestra.run.smithi055.stdout:smithi055.front.sepia.ceph.com
2021-12-12T19:15:28.684 INFO:teuthology.orchestra.run.smithi055.stdout:172.21.15.55 smithi055.front.sepia.ceph.com smithi055
2021-12-12T19:15:29.020 INFO:teuthology.provision.fog.smithi055:Deploy complete!
2021-12-12T19:15:42.340 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-12-12T19:15:42.692 INFO:teuthology.provision.fog.smithi090:Node is ready
2021-12-12T19:15:42.709 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090.front.sepia.ceph.com
2021-12-12T19:15:42.764 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90 smithi090.front.sepia.ceph.com smithi090
2021-12-12T19:15:43.097 INFO:teuthology.provision.fog.smithi090:Deploy complete!
2021-12-12T19:15:43.287 INFO:teuthology.lock.ops:Checking smithi050.front.sepia.ceph.com
2021-12-12T19:15:43.288 INFO:teuthology.lock.ops:New key found. Updating...
2021-12-12T19:15:43.314 INFO:teuthology.lock.ops:Checking smithi055.front.sepia.ceph.com
2021-12-12T19:15:43.315 INFO:teuthology.lock.ops:New key found. Updating...
2021-12-12T19:15:43.333 INFO:teuthology.lock.ops:Checking smithi090.front.sepia.ceph.com
2021-12-12T19:15:43.333 INFO:teuthology.lock.ops:New key found. Updating...
2021-12-12T19:15:43.352 INFO:teuthology.lock.ops:Updating [smithi050.front.sepia.ceph.com]: set os type and version on server
2021-12-12T19:15:43.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com'
2021-12-12T19:15:43.634 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64
2021-12-12T19:15:43.689 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Linux"
2021-12-12T19:15:43.690 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="8"
2021-12-12T19:15:43.690 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos"
2021-12-12T19:15:43.690 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora"
2021-12-12T19:15:43.691 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="8"
2021-12-12T19:15:43.691 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el8"
2021-12-12T19:15:43.691 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Linux 8"
2021-12-12T19:15:43.691 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31"
2021-12-12T19:15:43.692 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-12-12T19:15:43.692 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/"
2021-12-12T19:15:43.692 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-12-12T19:15:43.692 INFO:teuthology.orchestra.run.smithi050.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-12-12T19:15:43.692 INFO:teuthology.orchestra.run.smithi050.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-12-12T19:15:43.694 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server
2021-12-12T19:15:43.715 INFO:teuthology.lock.ops:Updating [smithi055.front.sepia.ceph.com]: set os type and version on server
2021-12-12T19:15:43.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com'
2021-12-12T19:15:43.932 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64
2021-12-12T19:15:43.993 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="CentOS Linux"
2021-12-12T19:15:43.994 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="8"
2021-12-12T19:15:43.995 INFO:teuthology.orchestra.run.smithi055.stdout:ID="centos"
2021-12-12T19:15:43.995 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE="rhel fedora"
2021-12-12T19:15:43.996 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="8"
2021-12-12T19:15:43.996 INFO:teuthology.orchestra.run.smithi055.stdout:PLATFORM_ID="platform:el8"
2021-12-12T19:15:43.997 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="CentOS Linux 8"
2021-12-12T19:15:43.998 INFO:teuthology.orchestra.run.smithi055.stdout:ANSI_COLOR="0;31"
2021-12-12T19:15:43.998 INFO:teuthology.orchestra.run.smithi055.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-12-12T19:15:43.998 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://centos.org/"
2021-12-12T19:15:43.999 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-12-12T19:15:43.999 INFO:teuthology.orchestra.run.smithi055.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-12-12T19:15:44.000 INFO:teuthology.orchestra.run.smithi055.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-12-12T19:15:44.001 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server
2021-12-12T19:15:44.029 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: set os type and version on server
2021-12-12T19:15:44.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2021-12-12T19:15:44.239 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64
2021-12-12T19:15:44.294 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Linux"
2021-12-12T19:15:44.295 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="8"
2021-12-12T19:15:44.295 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos"
2021-12-12T19:15:44.295 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora"
2021-12-12T19:15:44.295 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="8"
2021-12-12T19:15:44.296 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el8"
2021-12-12T19:15:44.296 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Linux 8"
2021-12-12T19:15:44.296 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31"
2021-12-12T19:15:44.296 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-12-12T19:15:44.297 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/"
2021-12-12T19:15:44.297 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-12-12T19:15:44.297 INFO:teuthology.orchestra.run.smithi090.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-12-12T19:15:44.297 INFO:teuthology.orchestra.run.smithi090.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-12-12T19:15:44.298 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server
2021-12-12T19:15:44.415 INFO:teuthology.dispatcher.supervisor:Running job 6559706
2021-12-12T19:15:44.415 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology -v --owner vshankar --archive /home/teuthworker/archive/vshankar-2021-12-12_16:30:32-fs-wip-vshankar-testing-20211212-193118-testing-default-smithi/6559706 --name vshankar-2021-12-12_16:30:32-fs-wip-vshankar-testing-20211212-193118-testing-default-smithi --description fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} -- /home/teuthworker/archive/vshankar-2021-12-12_16:30:32-fs-wip-vshankar-testing-20211212-193118-testing-default-smithi/6559706/orig.config.yaml
2021-12-12T19:15:44.419 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2021-12-12_16:30:32-fs-wip-vshankar-testing-20211212-193118-testing-default-smithi/6559706
2021-12-12T19:15:44.419 INFO:teuthology.dispatcher.supervisor:Job PID: 29268
2021-12-12T19:15:44.420 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-12-13T07:16:34.102 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-12-13T07:16:34.325 INFO:teuthology.kill:Killing Pids: {29268}
2021-12-13T07:16:35.494 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2021-12-13T07:16:35.495 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2021-12-13T07:16:35.495 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2021-12-13T07:16:35.495 INFO:teuthology.misc:Compressing logs...
2021-12-13T07:16:35.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com'
2021-12-13T07:16:35.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com'
2021-12-13T07:16:36.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2021-12-13T07:16:36.515 INFO:teuthology.misc:Compressing logs...
2021-12-13T07:16:36.785 INFO:teuthology.misc:Compressing logs...
2021-12-13T07:26:31.777 INFO:teuthology.kill:No teuthology processes running
2021-12-13T07:26:32.978 INFO:teuthology.kill:Nuking machines: ['smithi050', 'smithi055', 'smithi090']
2021-12-13T07:26:34.003 INFO:teuthology.kill:2021-12-13 07:26:34,002.002 INFO:teuthology.nuke:targets:
2021-12-13T07:26:34.004 INFO:teuthology.kill:  smithi050.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsXCVFynr6U2rn6ZUZC5l6CGtEEO8SaXnLC928yiQIgolSNPCOIQgJD6US8DDP2BgzTqNwXgdSEJeoQnUrqPvnpIHairYlaLuqesKYgsBYFBQsDFYc0irhd8P8t0IGIrc0mNNSm4zScbtT3e8nJQBtVg7nEctIgCohLFCfKiAJhj6Spm8q/DT7SRsqz8qM9sNX38Hfzk/r6kIPKKUl06PKdsqJUXBPmL/rUAAB6T0S/nHP3k4Jg4ZvS0OgyriS2nZOzmuALtBVVo6X1w3UC34wXcjC88893ipow88uNSyHP9M4kD0gSSsm1WvAo6ElQFtKZaBu7lnO13jYG+3ZTFxjGW/esRe9yCp5qiLAGkTDw2DgH2S4FsXvBmP34nuk3VB7Q8XylcQX/C1EbpNbD/VnkLHyOpIiulvkGhJT44BENfwcwA72ScsDy9HcgibQK/OoYHJwZr0QKDDALXtmq3T6wQceMSxgdxoRExzYsvLTTaIhKRZ11Vt+sQVKwEVARvk=
2021-12-13T07:26:34.004 INFO:teuthology.kill:  smithi055.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqUKLXgS0xQx0rqDsGxaRMaAR+Gd2FV7rJCo+LM4lWF9GVqHlX/aYF1NUU/uZxF//l7Eh3C5AN4jsrtKztDPRML5Z9UZXMUuU3KlNe+7pGgwutMOaTP/RDR0smNWI5ZGGvGtiwk5jIWJzUXyhdim21V+8JoluyPBNTbI/lQ5VRVlwWHrwL2bdC+HIbAM0VLCgPLRJNm3NkOgcnrWeQ80YfQiW3Fbf7a1CMEQDDCPTakE0koMJmMiVLDQnTlJRBUrlomnQlCZFFjxTBtJBr8tafJV1U1SGgkvxdB+j6dGvy97sjJpXGUz1GWqPE7TT8YcwqKfU7W/6Qs8FRJAFmE1miYcqXGbSJi4poU+eGNLVXXl2P5fdqvPRqKwLscFJduShpBai24CLhEdgBd+nQAFU/xY64kpntBKF7JJRexzJIPSkRMNYYMyEizXZaPqvEt1SEViOipwLDP6kD6NZsVLC6uFD5Ohp74K+RSh+aNHsaarzy+G9eIitr+pKXHPlaJys=
2021-12-13T07:26:34.004 INFO:teuthology.kill:  smithi090.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDqb+oqKNrvtaGObEUjQVMFVemRLv3mcfkRv9iKjpg684h//kXfFy5TCx2lnMAAirjGY8K3YD0G0uCRpdyR5+dsYnE2tANIqMchJHa0qlvncPwH6M/jEmx9DLxw+scKs7b0q1RukYuojbNmrsUGxJY1g/GdtxPQ/lK2brpU3FoyDcrpm4p1FjdTX3kmyqo1Sq+edL5frz8ZnN2VcmQ+YvLMPL6EWl6YqDXkY7whGe5ccKuvLo2NWU+6+c+pJtZe2IHobxh1KQRliarlMCoZOHM++Xy0AEbW6tE54/ey/Dvu8HbFPfv/Mnrn0mpDVAhNmNJPOPbVcwHcXBnRm+S4VOPDF72XLHPtSTunyWka+PUDO+qFa2pLCeLjKNH+VEb0p7NeI60LRPQr8i1ifLh2NSUfrg6KO93YuVuJhx7fiNY42WyZgWCCl8MjEAHuIOag1lpD4ZP/YH7okQn96gNTTuTmh9pCAVdOpoJaP/cTQXk6LmaGBpYNGPdhSJUPBHBnS0=
2021-12-13T07:26:34.033 INFO:teuthology.kill:2021-12-13 07:26:34,033.033 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-12-13T07:26:34.038 INFO:teuthology.kill:2021-12-13 07:26:34,038.038 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-12-13T07:26:34.040 INFO:teuthology.kill:2021-12-13 07:26:34,040.040 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-12-13T07:26:34.133 INFO:teuthology.kill:2021-12-13 07:26:34,133.133 INFO:teuthology.orchestra.console:Power off smithi090
2021-12-13T07:26:34.148 INFO:teuthology.kill:2021-12-13 07:26:34,147.147 INFO:teuthology.orchestra.console:Power off smithi055
2021-12-13T07:26:34.159 INFO:teuthology.kill:2021-12-13 07:26:34,159.159 INFO:teuthology.orchestra.console:Power off smithi050
2021-12-13T07:26:42.528 INFO:teuthology.kill:2021-12-13 07:26:42,528.528 INFO:teuthology.orchestra.console:Power off for smithi055 completed
2021-12-13T07:26:42.676 INFO:teuthology.kill:2021-12-13 07:26:42,675.675 INFO:teuthology.lock.ops:unlocked: smithi055.front.sepia.ceph.com
2021-12-13T07:26:46.634 INFO:teuthology.kill:2021-12-13 07:26:46,633.633 INFO:teuthology.orchestra.console:Power off for smithi090 completed
2021-12-13T07:26:46.645 INFO:teuthology.kill:2021-12-13 07:26:46,645.645 INFO:teuthology.orchestra.console:Power off for smithi050 completed
2021-12-13T07:26:46.784 INFO:teuthology.kill:2021-12-13 07:26:46,784.784 INFO:teuthology.lock.ops:unlocked: smithi090.front.sepia.ceph.com
2021-12-13T07:26:46.791 INFO:teuthology.kill:2021-12-13 07:26:46,791.791 INFO:teuthology.lock.ops:unlocked: smithi050.front.sepia.ceph.com
2021-12-13T07:28:47.155 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15