2022-01-12T19:27:02.426 INFO:root:teuthology version: 1.1.0-6fc20113
2022-01-12T19:27:02.555 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi110.front.sepia.ceph.com' reimaging
2022-01-12T19:27:02.556 INFO:teuthology.lock.ops:Updating [ubuntu@smithi110.front.sepia.ceph.com]: reset os type and version on server
2022-01-12T19:27:02.556 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server
2022-01-12T19:27:02.576 INFO:teuthology.lock.ops:Node 'ubuntu@smithi110.front.sepia.ceph.com' reimaging is complete
2022-01-12T19:27:02.576 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi145.front.sepia.ceph.com' reimaging
2022-01-12T19:27:02.576 INFO:teuthology.lock.ops:Updating [ubuntu@smithi145.front.sepia.ceph.com]: reset os type and version on server
2022-01-12T19:27:02.577 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server
2022-01-12T19:27:02.598 INFO:teuthology.lock.ops:Node 'ubuntu@smithi145.front.sepia.ceph.com' reimaging is complete
2022-01-12T19:27:02.795 INFO:teuthology.provision.fog.smithi110:Scheduling deploy of ubuntu 20.04
2022-01-12T19:27:02.810 INFO:teuthology.provision.fog.smithi145:Scheduling deploy of ubuntu 20.04
2022-01-12T19:27:03.156 INFO:teuthology.orchestra.console:Power off smithi110
2022-01-12T19:27:03.170 INFO:teuthology.orchestra.console:Power off smithi145
2022-01-12T19:27:11.531 INFO:teuthology.orchestra.console:Power off for smithi110 completed
2022-01-12T19:27:11.537 INFO:teuthology.orchestra.console:Power off for smithi145 completed
2022-01-12T19:27:11.634 INFO:teuthology.orchestra.console:Power on smithi110
2022-01-12T19:27:11.643 INFO:teuthology.orchestra.console:Power on smithi145
2022-01-12T19:27:15.905 INFO:teuthology.orchestra.console:Power on for smithi145 completed
2022-01-12T19:27:16.007 INFO:teuthology.provision.fog.smithi145:Waiting for deploy to finish
2022-01-12T19:27:20.003 INFO:teuthology.orchestra.console:Power on for smithi110 completed
2022-01-12T19:27:20.105 INFO:teuthology.provision.fog.smithi110:Waiting for deploy to finish
2022-01-12T19:30:21.317 ERROR:teuthology.orchestra.connection:Error authenticating with smithi110.front.sepia.ceph.com: Authentication failed.
2022-01-12T19:32:50.595 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-01-12T19:32:51.118 INFO:teuthology.provision.fog.smithi145:Node is ready
2022-01-12T19:32:51.125 INFO:teuthology.orchestra.run.smithi145.stdout:smithi145.front.sepia.ceph.com
2022-01-12T19:32:51.176 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145 smithi145.front.sepia.ceph.com smithi145
2022-01-12T19:32:51.322 INFO:teuthology.provision.fog.smithi145:Deploy complete!
2022-01-12T19:33:06.146 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-01-12T19:33:06.665 INFO:teuthology.provision.fog.smithi110:Node is ready
2022-01-12T19:33:06.671 INFO:teuthology.orchestra.run.smithi110.stdout:smithi110.front.sepia.ceph.com
2022-01-12T19:33:06.679 INFO:teuthology.orchestra.run.smithi110.stdout:172.21.15.110 smithi110.front.sepia.ceph.com smithi110
2022-01-12T19:33:06.902 INFO:teuthology.provision.fog.smithi110:Deploy complete!
2022-01-12T19:33:07.087 INFO:teuthology.lock.ops:Checking smithi110.front.sepia.ceph.com
2022-01-12T19:33:07.088 INFO:teuthology.lock.ops:New key found. Updating...
2022-01-12T19:33:07.110 INFO:teuthology.lock.ops:Checking smithi145.front.sepia.ceph.com
2022-01-12T19:33:07.110 INFO:teuthology.lock.ops:New key found. Updating...
2022-01-12T19:33:07.128 INFO:teuthology.lock.ops:Updating [smithi110.front.sepia.ceph.com]: set os type and version on server
2022-01-12T19:33:07.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com'
2022-01-12T19:33:07.302 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64
2022-01-12T19:33:07.353 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Ubuntu"
2022-01-12T19:33:07.353 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-01-12T19:33:07.353 INFO:teuthology.orchestra.run.smithi110.stdout:ID=ubuntu
2022-01-12T19:33:07.354 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE=debian
2022-01-12T19:33:07.354 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-01-12T19:33:07.354 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="20.04"
2022-01-12T19:33:07.354 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.ubuntu.com/"
2022-01-12T19:33:07.355 INFO:teuthology.orchestra.run.smithi110.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-01-12T19:33:07.355 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-01-12T19:33:07.355 INFO:teuthology.orchestra.run.smithi110.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-01-12T19:33:07.355 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_CODENAME=focal
2022-01-12T19:33:07.356 INFO:teuthology.orchestra.run.smithi110.stdout:UBUNTU_CODENAME=focal
2022-01-12T19:33:07.357 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server
2022-01-12T19:33:07.375 INFO:teuthology.lock.ops:Updating [smithi145.front.sepia.ceph.com]: set os type and version on server
2022-01-12T19:33:07.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com'
2022-01-12T19:33:07.546 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64
2022-01-12T19:33:07.595 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="Ubuntu"
2022-01-12T19:33:07.595 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-01-12T19:33:07.596 INFO:teuthology.orchestra.run.smithi145.stdout:ID=ubuntu
2022-01-12T19:33:07.596 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE=debian
2022-01-12T19:33:07.596 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-01-12T19:33:07.596 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="20.04"
2022-01-12T19:33:07.597 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://www.ubuntu.com/"
2022-01-12T19:33:07.597 INFO:teuthology.orchestra.run.smithi145.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-01-12T19:33:07.597 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-01-12T19:33:07.597 INFO:teuthology.orchestra.run.smithi145.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-01-12T19:33:07.597 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_CODENAME=focal
2022-01-12T19:33:07.598 INFO:teuthology.orchestra.run.smithi145.stdout:UBUNTU_CODENAME=focal
2022-01-12T19:33:07.598 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server
2022-01-12T19:33:07.698 INFO:teuthology.dispatcher.supervisor:Running job 6610648
2022-01-12T19:33:07.699 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_6fc2011361437a9dfe4e45b50de224392eed8abc/virtualenv/bin/teuthology -v --owner scheduled_cbodley@teuthology --archive /home/teuthworker/archive/cbodley-2022-01-12_19:23:19-rgw-wip-yuval-dynamic-reshard-distro-default-smithi/6610648 --name cbodley-2022-01-12_19:23:19-rgw-wip-yuval-dynamic-reshard-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{ubuntu_latest} tasks/test_multi} -- /home/teuthworker/archive/cbodley-2022-01-12_19:23:19-rgw-wip-yuval-dynamic-reshard-distro-default-smithi/6610648/orig.config.yaml
2022-01-12T19:33:07.703 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/cbodley-2022-01-12_19:23:19-rgw-wip-yuval-dynamic-reshard-distro-default-smithi/6610648
2022-01-12T19:33:07.703 INFO:teuthology.dispatcher.supervisor:Job PID: 26261
2022-01-12T19:33:07.704 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-01-13T07:33:27.561 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2022-01-13T07:33:27.744 INFO:teuthology.kill:Killing Pids: {26261}
2022-01-13T07:33:28.947 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1']
2022-01-13T07:33:28.948 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']
2022-01-13T07:33:28.949 INFO:teuthology.misc:Compressing logs...
2022-01-13T07:33:28.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com'
2022-01-13T07:33:29.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com'
2022-01-13T07:33:29.553 INFO:teuthology.misc:Compressing logs...
2022-01-13T07:33:29.688 INFO:teuthology.misc:Compressing logs...
2022-01-13T07:33:41.414 INFO:teuthology.kill:No teuthology processes running
2022-01-13T07:33:42.564 INFO:teuthology.kill:Nuking machines: ['smithi110', 'smithi145']
2022-01-13T07:33:43.583 INFO:teuthology.kill:2022-01-13 07:33:43,583.583 INFO:teuthology.nuke:targets:
2022-01-13T07:33:43.584 INFO:teuthology.kill:  smithi110.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUC+OyVxpfK/WRqXrd7COLEbJqFj4Rdux8ricdfClmZFztsUgdqVGffZLJLpwdh4rzJ2/LHNBCffVjYwQ5PfM6ijb3Eht7uHiaQ1XRYD/l4HxvQ2heyTUzBbqjafDU5HA4hLmvsYCAOZzS7/GHsPr0QBK9eIRsgaQQ9m0SJ2imnyNhk+zpFoEUtMlBl91z1NTrL8uA3R/9VDGMD4KzdOnPxhqPflrH3u+4yAK54WY/EiqM2ShEbnnrmOn4tg++AAFSIjG67NNHE1KpLWLSls9XAkGf4nXn/GXVTRh6B/y/6m6R7mElGreWbaUPex5pvhqfVczgaC0Xnx3vZKHWeWWoiFOF2GnQoQ8qWRB0X2AAxhVh4ULPpL7jRrvJq7VrAqStCiK4zT9h4g3yF8fJEIqWSYUCslBAIZoiRR2zOtwarbvodfDhikAQQ2p8U3g9EVQmVjjiBrTCyYcQMOKRoF/+Rf3wdfRTacTz8FsHgpXXDCgOGwcQnOw0DGsdv946Wn0=
2022-01-13T07:33:43.584 INFO:teuthology.kill:  smithi145.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJPCo0fMn6FyCizjB7d//RYaLw/rAf1e9A48lNrxrosektXzymNQL+437FJcO4YS9KK6fA12B7qTRuLmi45BTtNa8ZTUVB6N4/LPW1ZBfqSE6ZbOk3aNMUG9kRnsUSY111oNJM738RiklxhSBqr2m36iPrdCBF9cMdI+Hpa5Y7YWt5fWyQeVTKewAUwCkH3usu58XkCN413KifdQVT0n0Ea/mLFYKYg4JFmOJpQIib9+hd0KQtDwT9Qq9t9cwDKmcrgNdWMXMzshHxJTBzbV4xo4zvuHxQZuNVd3JiRfFRnwDFgOxRJBMXCxGgxeaanyG8yydXlj3zCuJ5RU9zBwszqI0+DkbIqYO8oh2Ykp12gwbyAgKlOWMTUG8V50JqpHGjkwW2Y0vTV1kzdugKitSwTuAAl+7MUWUSwLeSLos7VCcQKJskswaWq5bPs69YKeNnc2mMgAV+RJ0pUkJw0F7IMtqMGcBmBuDNCxesvNm+ppLlH8DkMmRdeR5yFeRGf0s=
2022-01-13T07:33:43.612 INFO:teuthology.kill:2022-01-13 07:33:43,611.611 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-01-13T07:33:43.615 INFO:teuthology.kill:2022-01-13 07:33:43,615.615 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-01-13T07:33:43.695 INFO:teuthology.kill:2022-01-13 07:33:43,695.695 INFO:teuthology.orchestra.console:Power off smithi145
2022-01-13T07:33:43.715 INFO:teuthology.kill:2022-01-13 07:33:43,714.714 INFO:teuthology.orchestra.console:Power off smithi110
2022-01-13T07:33:52.077 INFO:teuthology.kill:2022-01-13 07:33:52,077.077 INFO:teuthology.orchestra.console:Power off for smithi145 completed
2022-01-13T07:33:52.222 INFO:teuthology.kill:2022-01-13 07:33:52,221.221 INFO:teuthology.lock.ops:unlocked: smithi145.front.sepia.ceph.com
2022-01-13T07:33:56.200 INFO:teuthology.kill:2022-01-13 07:33:56,199.199 INFO:teuthology.orchestra.console:Power off for smithi110 completed
2022-01-13T07:33:56.354 INFO:teuthology.kill:2022-01-13 07:33:56,353.353 INFO:teuthology.lock.ops:unlocked: smithi110.front.sepia.ceph.com
2022-01-13T07:35:56.629 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-01-13T07:35:56.732 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi110 but it was locked by another job: /home/teuthworker/archive/benhanokh-2022-01-12_19:45:04-rados-WIP_GBH_NCB_expand_device_fix_1-distro-basic-smithi/6610830
2022-01-13T07:35:56.732 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi145 but it was locked by another job: /home/teuthworker/archive/benhanokh-2022-01-12_19:45:04-rados-WIP_GBH_NCB_expand_device_fix_1-distro-basic-smithi/6610830