2021-11-03T07:22:58.183 INFO:root:teuthology version: 1.1.0-c56135d1 2021-11-03T07:22:58.344 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi085.front.sepia.ceph.com' reimaging 2021-11-03T07:22:58.345 INFO:teuthology.lock.ops:Updating [ubuntu@smithi085.front.sepia.ceph.com]: reset os type and version on server 2021-11-03T07:22:58.345 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2021-11-03T07:22:58.364 INFO:teuthology.lock.ops:Node 'ubuntu@smithi085.front.sepia.ceph.com' reimaging is complete 2021-11-03T07:22:58.365 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi135.front.sepia.ceph.com' reimaging 2021-11-03T07:22:58.365 INFO:teuthology.lock.ops:Updating [ubuntu@smithi135.front.sepia.ceph.com]: reset os type and version on server 2021-11-03T07:22:58.366 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2021-11-03T07:22:58.386 INFO:teuthology.lock.ops:Node 'ubuntu@smithi135.front.sepia.ceph.com' reimaging is complete 2021-11-03T07:22:58.387 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi140.front.sepia.ceph.com' reimaging 2021-11-03T07:22:58.387 INFO:teuthology.lock.ops:Updating [ubuntu@smithi140.front.sepia.ceph.com]: reset os type and version on server 2021-11-03T07:22:58.388 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server 2021-11-03T07:22:58.405 INFO:teuthology.lock.ops:Node 'ubuntu@smithi140.front.sepia.ceph.com' reimaging is complete 2021-11-03T07:22:58.596 INFO:teuthology.provision.fog.smithi085:Scheduling deploy of ubuntu 20.04 2021-11-03T07:22:58.620 INFO:teuthology.provision.fog.smithi135:Scheduling deploy of ubuntu 20.04 2021-11-03T07:22:58.654 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of ubuntu 20.04 2021-11-03T07:22:58.978 INFO:teuthology.orchestra.console:Power off smithi135 2021-11-03T07:22:58.995 INFO:teuthology.orchestra.console:Power off smithi085 2021-11-03T07:22:59.032 INFO:teuthology.orchestra.console:Power off smithi140 2021-11-03T07:23:07.370 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2021-11-03T07:23:07.471 INFO:teuthology.orchestra.console:Power on smithi085 2021-11-03T07:23:11.477 INFO:teuthology.orchestra.console:Power off for smithi135 completed 2021-11-03T07:23:11.512 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2021-11-03T07:23:11.579 INFO:teuthology.orchestra.console:Power on smithi135 2021-11-03T07:23:11.614 INFO:teuthology.orchestra.console:Power on smithi140 2021-11-03T07:23:15.836 INFO:teuthology.orchestra.console:Power on for smithi085 completed 2021-11-03T07:23:15.844 INFO:teuthology.orchestra.console:Power on for smithi135 completed 2021-11-03T07:23:15.864 INFO:teuthology.orchestra.console:Power on for smithi140 completed 2021-11-03T07:23:15.938 INFO:teuthology.provision.fog.smithi085:Waiting for deploy to finish 2021-11-03T07:23:15.944 INFO:teuthology.provision.fog.smithi135:Waiting for deploy to finish 2021-11-03T07:23:15.965 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish 2021-11-03T07:28:35.814 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-03T07:28:36.360 INFO:teuthology.provision.fog.smithi085:Node is ready 2021-11-03T07:28:36.367 INFO:teuthology.orchestra.run.smithi085.stdout:smithi085.front.sepia.ceph.com 2021-11-03T07:28:36.418 INFO:teuthology.orchestra.run.smithi085.stdout:172.21.15.85 smithi085.front.sepia.ceph.com smithi085 2021-11-03T07:28:36.595 INFO:teuthology.provision.fog.smithi085:Deploy complete! 2021-11-03T07:28:47.716 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-03T07:28:48.248 INFO:teuthology.provision.fog.smithi140:Node is ready 2021-11-03T07:28:48.254 INFO:teuthology.orchestra.run.smithi140.stdout:smithi140.front.sepia.ceph.com 2021-11-03T07:28:48.307 INFO:teuthology.orchestra.run.smithi140.stdout:172.21.15.140 smithi140.front.sepia.ceph.com smithi140 2021-11-03T07:28:48.526 INFO:teuthology.provision.fog.smithi140:Deploy complete! 2021-11-03T07:28:50.853 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-03T07:28:51.345 INFO:teuthology.provision.fog.smithi135:Node is ready 2021-11-03T07:28:51.352 INFO:teuthology.orchestra.run.smithi135.stdout:smithi135.front.sepia.ceph.com 2021-11-03T07:28:51.359 INFO:teuthology.orchestra.run.smithi135.stdout:172.21.15.135 smithi135.front.sepia.ceph.com smithi135 2021-11-03T07:28:51.567 INFO:teuthology.provision.fog.smithi135:Deploy complete! 2021-11-03T07:28:51.829 INFO:teuthology.lock.ops:Checking smithi085.front.sepia.ceph.com 2021-11-03T07:28:51.830 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-03T07:28:51.849 INFO:teuthology.lock.ops:Checking smithi135.front.sepia.ceph.com 2021-11-03T07:28:51.849 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-03T07:28:51.867 INFO:teuthology.lock.ops:Checking smithi140.front.sepia.ceph.com 2021-11-03T07:28:51.867 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-03T07:28:51.886 INFO:teuthology.lock.ops:Updating [smithi085.front.sepia.ceph.com]: set os type and version on server 2021-11-03T07:28:51.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com' 2021-11-03T07:28:52.198 INFO:teuthology.orchestra.run.smithi085.stdout:x86_64 2021-11-03T07:28:52.205 INFO:teuthology.orchestra.run.smithi085.stdout:NAME="Ubuntu" 2021-11-03T07:28:52.205 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2021-11-03T07:28:52.205 INFO:teuthology.orchestra.run.smithi085.stdout:ID=ubuntu 2021-11-03T07:28:52.206 INFO:teuthology.orchestra.run.smithi085.stdout:ID_LIKE=debian 2021-11-03T07:28:52.206 INFO:teuthology.orchestra.run.smithi085.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2021-11-03T07:28:52.206 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_ID="20.04" 2021-11-03T07:28:52.206 INFO:teuthology.orchestra.run.smithi085.stdout:HOME_URL="https://www.ubuntu.com/" 2021-11-03T07:28:52.206 INFO:teuthology.orchestra.run.smithi085.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-11-03T07:28:52.207 INFO:teuthology.orchestra.run.smithi085.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-11-03T07:28:52.207 INFO:teuthology.orchestra.run.smithi085.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-11-03T07:28:52.207 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_CODENAME=focal 2021-11-03T07:28:52.207 INFO:teuthology.orchestra.run.smithi085.stdout:UBUNTU_CODENAME=focal 2021-11-03T07:28:52.208 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2021-11-03T07:28:52.226 INFO:teuthology.lock.ops:Updating [smithi135.front.sepia.ceph.com]: set os type and version on server 2021-11-03T07:28:52.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2021-11-03T07:28:52.398 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2021-11-03T07:28:52.407 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="Ubuntu" 2021-11-03T07:28:52.407 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2021-11-03T07:28:52.407 INFO:teuthology.orchestra.run.smithi135.stdout:ID=ubuntu 2021-11-03T07:28:52.407 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE=debian 2021-11-03T07:28:52.408 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2021-11-03T07:28:52.408 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="20.04" 2021-11-03T07:28:52.408 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="https://www.ubuntu.com/" 2021-11-03T07:28:52.408 INFO:teuthology.orchestra.run.smithi135.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-11-03T07:28:52.409 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-11-03T07:28:52.409 INFO:teuthology.orchestra.run.smithi135.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-11-03T07:28:52.409 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_CODENAME=focal 2021-11-03T07:28:52.409 INFO:teuthology.orchestra.run.smithi135.stdout:UBUNTU_CODENAME=focal 2021-11-03T07:28:52.410 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2021-11-03T07:28:52.427 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: set os type and version on server 2021-11-03T07:28:52.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com' 2021-11-03T07:28:52.604 INFO:teuthology.orchestra.run.smithi140.stdout:x86_64 2021-11-03T07:28:52.609 INFO:teuthology.orchestra.run.smithi140.stdout:NAME="Ubuntu" 2021-11-03T07:28:52.610 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2021-11-03T07:28:52.610 INFO:teuthology.orchestra.run.smithi140.stdout:ID=ubuntu 2021-11-03T07:28:52.610 INFO:teuthology.orchestra.run.smithi140.stdout:ID_LIKE=debian 2021-11-03T07:28:52.611 INFO:teuthology.orchestra.run.smithi140.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2021-11-03T07:28:52.611 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION_ID="20.04" 2021-11-03T07:28:52.611 INFO:teuthology.orchestra.run.smithi140.stdout:HOME_URL="https://www.ubuntu.com/" 2021-11-03T07:28:52.611 INFO:teuthology.orchestra.run.smithi140.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-11-03T07:28:52.612 INFO:teuthology.orchestra.run.smithi140.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-11-03T07:28:52.612 INFO:teuthology.orchestra.run.smithi140.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-11-03T07:28:52.612 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION_CODENAME=focal 2021-11-03T07:28:52.612 INFO:teuthology.orchestra.run.smithi140.stdout:UBUNTU_CODENAME=focal 2021-11-03T07:28:52.613 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server 2021-11-03T07:28:52.707 INFO:teuthology.dispatcher.supervisor:Running job 6480151 2021-11-03T07:28:52.708 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology -v --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2021-11-03_05:35:37-fs-wip-pdonnell-testing-20211103.023355-distro-basic-smithi/6480151 --name pdonnell-2021-11-03_05:35:37-fs-wip-pdonnell-testing-20211103.023355-distro-basic-smithi --description fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} -- /home/teuthworker/archive/pdonnell-2021-11-03_05:35:37-fs-wip-pdonnell-testing-20211103.023355-distro-basic-smithi/6480151/orig.config.yaml 2021-11-03T07:28:52.711 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/pdonnell-2021-11-03_05:35:37-fs-wip-pdonnell-testing-20211103.023355-distro-basic-smithi/6480151 2021-11-03T07:28:52.712 INFO:teuthology.dispatcher.supervisor:Job PID: 17163 2021-11-03T07:28:52.712 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-11-03T19:29:12.966 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-11-03T19:29:13.193 INFO:teuthology.kill:Killing Pids: {17163} 2021-11-03T19:29:14.422 INFO:teuthology.task.internal:roles: ubuntu@smithi085.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2021-11-03T19:29:14.423 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2021-11-03T19:29:14.424 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2021-11-03T19:29:14.424 INFO:teuthology.misc:Compressing logs... 2021-11-03T19:29:14.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com' 2021-11-03T19:29:14.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2021-11-03T19:29:15.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com' 2021-11-03T19:29:15.399 INFO:teuthology.misc:Compressing logs... 2021-11-03T19:29:15.626 INFO:teuthology.misc:Compressing logs... 2021-11-03T19:36:38.362 INFO:teuthology.kill:No teuthology processes running 2021-11-03T19:36:39.512 INFO:teuthology.kill:Nuking machines: ['smithi085', 'smithi135', 'smithi140'] 2021-11-03T19:36:40.477 INFO:teuthology.kill:2021-11-03 19:36:40,476.476 INFO:teuthology.nuke:targets: 2021-11-03T19:36:40.478 INFO:teuthology.kill: smithi085.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmknTILibBuXvOeSPHtL0YLDj6Yj8h57XYIK6k6uYNShDUNEzu45clfXbjb1Eq1yh2o57Yq3tnVCyk43/RdV0w437bom4Z9EUz1859kR68xrBCMlEaVv/NxJ/PgxjhSQoaHkhmeN1bLu1HAA2x0hQayM61gxMKzZsTEwrhR3+CcbG0ZRF6PKlgtokmvrQqtKueZNAodqpyOIeXUEgE/jOIEX8yVgvhcZ2aq7A3c6cTTNNzeu4Kf6mTamTWBxs6l85YEpOYUuj3ZviHdgjndTVlT6ceO0aIb0KiKH8RucjOXxlhA8JIxpLqrtwP1VoX2bdMILueeRyFpghZAwlpt7DlTbc8vK3aWiUYaFleUqx6UE6fM8/dMxE69d1Q5ID9EqgRL+QAbkNBBc1lsXlcrEdScLqSVLVOZHMdr9HAGtJF2DV7drCxjMMr2JL7f9kjhI5RvKZ60jscVB2HNSv6qUwbGCC8/YToQdy/8sI45eYOAfbnfN58z0Zkt5054u/U+TM= 2021-11-03T19:36:40.478 INFO:teuthology.kill: smithi135.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDvFlNxpJD8UM8L+D3mOPJv04oJxp/r0RwbTGDoa5RH7CCBwdbxNOCV8LA4N7UQLvDq+49CUW1ICCc/L72uvMOYty3Ie+nRgGjjN6Y5O6wgEQs1JRukZZg8EvCAwBCsau+9qXfg8ql8NgsKkv7YIZpYsgrYdnJXPpFTb4aKlahPw2S2v9XDMIUvqrMFwORgHBCUO0CyCBMWelYEv7Cki/hjXkHedWzfbNhbfbXNnuaiVFq55uiubgoqPz4YMNTj2j1eUes5mfIqlDwmGWRifMhBHNcRw8bhvPm2TLr9vDZvesoWZNMHz2vfHl5upoSuCiKVNFznI9wD7KwWdNT9RapLQ88Zx1yUd9Xmn8DJmpzpAXccNhDB2vvxRkQjAH014PfAuTXoMtMqj+7GWKB1CjGME1h4CNz+P7O+E9jN9xBSV7v3Y/4fX5d7PBypjkHyCW/h4+J/60LyFQMbJnf4nPXJmnnOrDzFQt51KqSqGqwYkoxjywV7yqzWo650Y+wjb80= 2021-11-03T19:36:40.479 INFO:teuthology.kill: smithi140.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDyAyHVHW9MPt7h6Cy8hgNLwySfaVKV9oP+TQOuKUiePaaRFLR1ZxxOdoGqxzOwEnel8T01OG1ziLoJ/jY2YlDKMeblLc0YLw45krKIG9gsxQ+5vIpJV+JYNAOHjbFBMhTLIUnxjlkOvl7xt2W8SephTD+iS47Wl1g2PjMCFROWNwHKUi8nk045hC1p3PwHdyyePGn98PN5WNsDmafiG3gQJ1T6TRTMV8SecCJmtH5g1AV1XZ3CVXF4U4GYQg1ULoOyWNNIusmrYTTwXytl7wMT49+fwgegWF7YaQ1yQlx06MWotZ+9qYJrV5Y09B/BbLs5o+SaCzWzLGfT0DJrgaYvWm22kX3/dVcKaLxktcTUDoxkitvUMGHplimjtpkTMw3kdSuQunKLL+GOewsCnSbLi28ZOwjaQlVc6GXDwLy0Gk+H02k09AHVFPPOBC3MFB3HMVGDsE5pGELTewg2cVZPG+1gb4WR69xFlzaRxFmlUmjOmqs7toYC/vciG48EmqU= 2021-11-03T19:36:40.507 INFO:teuthology.kill:2021-11-03 19:36:40,506.506 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-03T19:36:40.511 INFO:teuthology.kill:2021-11-03 19:36:40,511.511 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-03T19:36:40.512 INFO:teuthology.kill:2021-11-03 19:36:40,512.512 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-03T19:36:40.593 INFO:teuthology.kill:2021-11-03 19:36:40,593.593 INFO:teuthology.orchestra.console:Power off smithi140 2021-11-03T19:36:40.608 INFO:teuthology.kill:2021-11-03 19:36:40,608.608 INFO:teuthology.orchestra.console:Power off smithi085 2021-11-03T19:36:40.620 INFO:teuthology.kill:2021-11-03 19:36:40,619.619 INFO:teuthology.orchestra.console:Power off smithi135 2021-11-03T19:36:48.982 INFO:teuthology.kill:2021-11-03 19:36:48,982.982 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2021-11-03T19:36:49.129 INFO:teuthology.kill:2021-11-03 19:36:49,128.128 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com 2021-11-03T19:36:53.090 INFO:teuthology.kill:2021-11-03 19:36:53,089.089 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2021-11-03T19:36:53.098 INFO:teuthology.kill:2021-11-03 19:36:53,097.097 INFO:teuthology.orchestra.console:Power off for smithi135 completed 2021-11-03T19:36:53.238 INFO:teuthology.kill:2021-11-03 19:36:53,238.238 INFO:teuthology.lock.ops:unlocked: smithi085.front.sepia.ceph.com 2021-11-03T19:36:53.241 INFO:teuthology.kill:2021-11-03 19:36:53,241.241 INFO:teuthology.lock.ops:unlocked: smithi135.front.sepia.ceph.com 2021-11-03T19:38:53.587 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-11-03T19:38:53.715 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi085 but it was locked by another job: /home/teuthworker/archive/sage-2021-11-03_18:51:05-orch:cephadm-wip-sage-testing-2021-11-03-1245-distro-basic-smithi/6482079 2021-11-03T19:38:53.715 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi135 but it was locked by another job: /home/teuthworker/archive/sage-2021-11-03_18:51:05-orch:cephadm-wip-sage-testing-2021-11-03-1245-distro-basic-smithi/6482079 2021-11-03T19:38:53.716 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi140 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-03_18:53:10-rados-wip-yuri5-testing-2021-11-02-0745-distro-basic-smithi/6482137