2022-03-03T21:55:58.282 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-03T21:55:58.402 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi122.front.sepia.ceph.com' reimaging 2022-03-03T21:55:58.403 INFO:teuthology.lock.ops:Updating [ubuntu@smithi122.front.sepia.ceph.com]: reset os type and version on server 2022-03-03T21:55:58.404 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2022-03-03T21:55:58.422 INFO:teuthology.lock.ops:Node 'ubuntu@smithi122.front.sepia.ceph.com' reimaging is complete 2022-03-03T21:55:58.423 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi197.front.sepia.ceph.com' reimaging 2022-03-03T21:55:58.423 INFO:teuthology.lock.ops:Updating [ubuntu@smithi197.front.sepia.ceph.com]: reset os type and version on server 2022-03-03T21:55:58.424 INFO:teuthology.lock.ops:Updating smithi197.front.sepia.ceph.com on lock server 2022-03-03T21:55:58.442 INFO:teuthology.lock.ops:Node 'ubuntu@smithi197.front.sepia.ceph.com' reimaging is complete 2022-03-03T21:55:58.637 INFO:teuthology.provision.fog.smithi122:Scheduling deploy of ubuntu 20.04 2022-03-03T21:55:58.655 INFO:teuthology.provision.fog.smithi197:Scheduling deploy of ubuntu 20.04 2022-03-03T21:55:59.102 INFO:teuthology.orchestra.console:Power off smithi122 2022-03-03T21:55:59.164 INFO:teuthology.orchestra.console:Power off smithi197 2022-03-03T21:56:11.583 INFO:teuthology.orchestra.console:Power off for smithi122 completed 2022-03-03T21:56:11.637 INFO:teuthology.orchestra.console:Power off for smithi197 completed 2022-03-03T21:56:11.685 INFO:teuthology.orchestra.console:Power on smithi122 2022-03-03T21:56:11.739 INFO:teuthology.orchestra.console:Power on smithi197 2022-03-03T21:56:15.942 INFO:teuthology.orchestra.console:Power on for smithi122 completed 2022-03-03T21:56:15.991 INFO:teuthology.orchestra.console:Power on for smithi197 completed 2022-03-03T21:56:16.044 INFO:teuthology.provision.fog.smithi122:Waiting for deploy to finish 2022-03-03T21:56:16.092 INFO:teuthology.provision.fog.smithi197:Waiting for deploy to finish 2022-03-03T22:01:24.669 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-03T22:01:25.148 INFO:teuthology.provision.fog.smithi122:Node is ready 2022-03-03T22:01:25.154 INFO:teuthology.orchestra.run.smithi122.stdout:smithi122.front.sepia.ceph.com 2022-03-03T22:01:25.162 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.15.122 smithi122.front.sepia.ceph.com smithi122 2022-03-03T22:01:25.362 INFO:teuthology.provision.fog.smithi122:Deploy complete! 2022-03-03T22:01:25.700 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-03T22:01:26.245 INFO:teuthology.provision.fog.smithi197:Node is ready 2022-03-03T22:01:26.252 INFO:teuthology.orchestra.run.smithi197.stdout:smithi197.front.sepia.ceph.com 2022-03-03T22:01:26.303 INFO:teuthology.orchestra.run.smithi197.stdout:172.21.15.197 smithi197.front.sepia.ceph.com smithi197 2022-03-03T22:01:26.476 INFO:teuthology.provision.fog.smithi197:Deploy complete! 2022-03-03T22:01:26.652 INFO:teuthology.lock.ops:Checking smithi122.front.sepia.ceph.com 2022-03-03T22:01:26.653 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-03T22:01:26.672 INFO:teuthology.lock.ops:Checking smithi197.front.sepia.ceph.com 2022-03-03T22:01:26.673 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-03T22:01:26.690 INFO:teuthology.lock.ops:Updating [smithi122.front.sepia.ceph.com]: set os type and version on server 2022-03-03T22:01:26.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2022-03-03T22:01:26.870 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2022-03-03T22:01:26.875 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="Ubuntu" 2022-03-03T22:01:26.876 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-03T22:01:26.876 INFO:teuthology.orchestra.run.smithi122.stdout:ID=ubuntu 2022-03-03T22:01:26.876 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE=debian 2022-03-03T22:01:26.876 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-03T22:01:26.877 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="20.04" 2022-03-03T22:01:26.877 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-03T22:01:26.877 INFO:teuthology.orchestra.run.smithi122.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-03T22:01:26.877 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-03T22:01:26.877 INFO:teuthology.orchestra.run.smithi122.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-03T22:01:26.878 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_CODENAME=focal 2022-03-03T22:01:26.878 INFO:teuthology.orchestra.run.smithi122.stdout:UBUNTU_CODENAME=focal 2022-03-03T22:01:26.879 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2022-03-03T22:01:26.897 INFO:teuthology.lock.ops:Updating [smithi197.front.sepia.ceph.com]: set os type and version on server 2022-03-03T22:01:26.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi197.front.sepia.ceph.com' 2022-03-03T22:01:27.078 INFO:teuthology.orchestra.run.smithi197.stdout:x86_64 2022-03-03T22:01:27.084 INFO:teuthology.orchestra.run.smithi197.stdout:NAME="Ubuntu" 2022-03-03T22:01:27.084 INFO:teuthology.orchestra.run.smithi197.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-03T22:01:27.085 INFO:teuthology.orchestra.run.smithi197.stdout:ID=ubuntu 2022-03-03T22:01:27.085 INFO:teuthology.orchestra.run.smithi197.stdout:ID_LIKE=debian 2022-03-03T22:01:27.085 INFO:teuthology.orchestra.run.smithi197.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-03T22:01:27.086 INFO:teuthology.orchestra.run.smithi197.stdout:VERSION_ID="20.04" 2022-03-03T22:01:27.086 INFO:teuthology.orchestra.run.smithi197.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-03T22:01:27.086 INFO:teuthology.orchestra.run.smithi197.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-03T22:01:27.086 INFO:teuthology.orchestra.run.smithi197.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-03T22:01:27.087 INFO:teuthology.orchestra.run.smithi197.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-03T22:01:27.087 INFO:teuthology.orchestra.run.smithi197.stdout:VERSION_CODENAME=focal 2022-03-03T22:01:27.088 INFO:teuthology.orchestra.run.smithi197.stdout:UBUNTU_CODENAME=focal 2022-03-03T22:01:27.089 INFO:teuthology.lock.ops:Updating smithi197.front.sepia.ceph.com on lock server 2022-03-03T22:01:27.181 INFO:teuthology.dispatcher.supervisor:Running job 6717529 2022-03-03T22:01:27.182 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-03-03_05:07:02-rgw-pacific-distro-default-smithi/6717529 --name teuthology-2022-03-03_05:07:02-rgw-pacific-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} -- /home/teuthworker/archive/teuthology-2022-03-03_05:07:02-rgw-pacific-distro-default-smithi/6717529/orig.config.yaml 2022-03-03T22:01:27.185 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-03-03_05:07:02-rgw-pacific-distro-default-smithi/6717529 2022-03-03T22:01:27.186 INFO:teuthology.dispatcher.supervisor:Job PID: 11642 2022-03-03T22:01:27.187 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-03-04T04:31:57.679 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-03-04T04:31:57.830 INFO:teuthology.kill:Killing Pids: {11642} 2022-03-04T04:31:59.192 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2022-03-04T04:31:59.194 INFO:teuthology.task.internal:roles: ubuntu@smithi197.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2022-03-04T04:31:59.194 INFO:teuthology.misc:Compressing logs... 2022-03-04T04:31:59.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2022-03-04T04:31:59.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi197.front.sepia.ceph.com' 2022-03-04T04:31:59.758 INFO:teuthology.misc:Compressing logs... 2022-03-04T04:31:59.892 INFO:teuthology.misc:Compressing logs... 2022-03-04T04:33:59.425 INFO:teuthology.kill:No teuthology processes running 2022-03-04T04:34:00.637 INFO:teuthology.kill:Nuking machines: ['smithi122', 'smithi197'] 2022-03-04T04:34:01.617 INFO:teuthology.kill:2022-03-04 04:34:01,616.616 INFO:teuthology.nuke:targets: 2022-03-04T04:34:01.618 INFO:teuthology.kill: smithi122.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDo8ctZzh3QtVIO1ajB6BiVsYMDT7hIMdOMA6wSPUHdE+DH40K3PPcLZ8lvAtHT3ENtv1mDOg3YiC7ARc5wbvYUvUD1YkJnK7piGofmWUytDSYo1/dHKthNUw5/9H/8rshzZqJD98jhh7he7uyYy4CgUqo2caDl2MpNPPl7q+IAcmLGcbXfmL/QN16vx2hdKtWwNg/dNCymeLWJV168dmdUHj2WvfRHjBAx00mNFkw8GaGEUFFKbap1YnPe/mc2dugGsAku4CbTmtEe0NGGpAhM1HdM4bzB84Hn6Us/5ipL7ThbTq9z/aoioun379eTdzNOug/PLe1S1s5FEhJma4tGGhsFkNf01ebBCpxmkivDI10NebT/KJgjcEBcn+nS3M86MGQumacvqpT6L0lHTb4SJnPVZLnnHRH24P9jwG9CyWPjOQsxrH4FZTkSOmqXI1T7/vkVaQwhHD3IckBFy4+JzArhCPPMCy+ehA7ef9QXSPswHD/hjpJ10P6bS1CP0zM= 2022-03-04T04:34:01.618 INFO:teuthology.kill: smithi197.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPyYzWFzY+3ppzJXF38uUwlx5aGxQs2uDGOYWSJNGI0cJUI9kUJIyGUaqHrFYuag/zduhG25N6wl5tkDMpcIQTvhwotJbWPbMgIP3Xnj+yLh+wBsYEUwtpvNSg7yfDKWVQCErh9HZULR5MAsbh3cIs9MO2MYPd6YvqNaxTHzbl0Q+n7badovvGdpTYNyK1iEsq2T2gyyMFIw5wCtgCjdY6nQlCMYAy1SAOdAPzI0F+eyofBnVsDDSOqYKgieu7qaTk//vJ0wEq5C+JrZpaakv05A8pKhAv/RZB4/1cf9XqG0aDVv+HkB2HCHwCaTQV1/loCgYiHgU+/GlUzZ+BezEAF+/+xh1ZvtMclUrEry+sjritj4gMLIvbFLk+Mdn0ysVvib5nb5C66aVDoXmWaDIdwyvk0WLrtwx7mTsgDxBALXtXqWa2qGNEkVl/KcLDyGUazviouAlCkDtTp6j0daXDqFQcdtKz/5NHrInI8cNOFySHVqiPw2nm/UgMoWo+ldc= 2022-03-04T04:34:01.652 INFO:teuthology.kill:2022-03-04 04:34:01,652.652 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-04T04:34:01.656 INFO:teuthology.kill:2022-03-04 04:34:01,656.656 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-04T04:34:01.749 INFO:teuthology.kill:2022-03-04 04:34:01,748.748 INFO:teuthology.orchestra.console:Power off smithi197 2022-03-04T04:34:01.765 INFO:teuthology.kill:2022-03-04 04:34:01,765.765 INFO:teuthology.orchestra.console:Power off smithi122 2022-03-04T04:34:14.237 INFO:teuthology.kill:2022-03-04 04:34:14,236.236 INFO:teuthology.orchestra.console:Power off for smithi197 completed 2022-03-04T04:34:14.244 INFO:teuthology.kill:2022-03-04 04:34:14,244.244 INFO:teuthology.orchestra.console:Power off for smithi122 completed 2022-03-04T04:34:14.399 INFO:teuthology.kill:2022-03-04 04:34:14,399.399 INFO:teuthology.lock.ops:unlocked: smithi197.front.sepia.ceph.com 2022-03-04T04:34:14.401 INFO:teuthology.kill:2022-03-04 04:34:14,401.401 INFO:teuthology.lock.ops:unlocked: smithi122.front.sepia.ceph.com 2022-03-04T04:36:14.745 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-03-04T04:36:14.865 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi122 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-02-27_05:15:02-krbd-nautilus-testing-default-smithi/6708876