2022-03-10T10:04:26.830 INFO:root:teuthology version: 1.1.0-ff5db916 2022-03-10T10:04:26.949 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi025.front.sepia.ceph.com' reimaging 2022-03-10T10:04:26.950 INFO:teuthology.lock.ops:Updating [ubuntu@smithi025.front.sepia.ceph.com]: reset os type and version on server 2022-03-10T10:04:26.950 INFO:teuthology.lock.ops:Updating smithi025.front.sepia.ceph.com on lock server 2022-03-10T10:04:26.969 INFO:teuthology.lock.ops:Node 'ubuntu@smithi025.front.sepia.ceph.com' reimaging is complete 2022-03-10T10:04:26.970 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi153.front.sepia.ceph.com' reimaging 2022-03-10T10:04:26.970 INFO:teuthology.lock.ops:Updating [ubuntu@smithi153.front.sepia.ceph.com]: reset os type and version on server 2022-03-10T10:04:26.971 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2022-03-10T10:04:26.989 INFO:teuthology.lock.ops:Node 'ubuntu@smithi153.front.sepia.ceph.com' reimaging is complete 2022-03-10T10:04:27.171 INFO:teuthology.provision.fog.smithi025:Scheduling deploy of ubuntu 20.04 2022-03-10T10:04:27.232 INFO:teuthology.provision.fog.smithi153:Scheduling deploy of ubuntu 20.04 2022-03-10T10:04:27.504 INFO:teuthology.orchestra.console:Power off smithi025 2022-03-10T10:04:27.621 INFO:teuthology.orchestra.console:Power off smithi153 2022-03-10T10:04:35.877 INFO:teuthology.orchestra.console:Power off for smithi025 completed 2022-03-10T10:04:35.979 INFO:teuthology.orchestra.console:Power on smithi025 2022-03-10T10:04:35.988 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2022-03-10T10:04:36.089 INFO:teuthology.orchestra.console:Power on smithi153 2022-03-10T10:04:44.415 INFO:teuthology.orchestra.console:Power on for smithi025 completed 2022-03-10T10:04:44.454 INFO:teuthology.orchestra.console:Power on for smithi153 completed 2022-03-10T10:04:44.517 INFO:teuthology.provision.fog.smithi025:Waiting for deploy to finish 2022-03-10T10:04:44.555 INFO:teuthology.provision.fog.smithi153:Waiting for deploy to finish 2022-03-10T10:07:46.076 ERROR:teuthology.orchestra.connection:Error authenticating with smithi153.front.sepia.ceph.com: Authentication failed. 2022-03-10T10:10:05.279 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-10T10:10:05.735 INFO:teuthology.provision.fog.smithi153:Node is ready 2022-03-10T10:10:05.742 INFO:teuthology.orchestra.run.smithi153.stdout:smithi153.front.sepia.ceph.com 2022-03-10T10:10:05.791 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153 smithi153.front.sepia.ceph.com smithi153 2022-03-10T10:10:06.011 INFO:teuthology.provision.fog.smithi153:Deploy complete! 2022-03-10T10:10:37.853 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-10T10:10:38.386 INFO:teuthology.provision.fog.smithi025:Node is ready 2022-03-10T10:10:38.394 INFO:teuthology.orchestra.run.smithi025.stdout:smithi025.front.sepia.ceph.com 2022-03-10T10:10:38.444 INFO:teuthology.orchestra.run.smithi025.stdout:172.21.15.25 smithi025.front.sepia.ceph.com smithi025 2022-03-10T10:10:38.621 INFO:teuthology.provision.fog.smithi025:Deploy complete! 2022-03-10T10:10:38.808 INFO:teuthology.lock.ops:Checking smithi025.front.sepia.ceph.com 2022-03-10T10:10:38.809 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-10T10:10:38.827 INFO:teuthology.lock.ops:Checking smithi153.front.sepia.ceph.com 2022-03-10T10:10:38.828 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-10T10:10:38.846 INFO:teuthology.lock.ops:Updating [smithi025.front.sepia.ceph.com]: set os type and version on server 2022-03-10T10:10:38.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi025.front.sepia.ceph.com' 2022-03-10T10:10:39.031 INFO:teuthology.orchestra.run.smithi025.stdout:x86_64 2022-03-10T10:10:39.038 INFO:teuthology.orchestra.run.smithi025.stdout:NAME="Ubuntu" 2022-03-10T10:10:39.039 INFO:teuthology.orchestra.run.smithi025.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-10T10:10:39.039 INFO:teuthology.orchestra.run.smithi025.stdout:ID=ubuntu 2022-03-10T10:10:39.040 INFO:teuthology.orchestra.run.smithi025.stdout:ID_LIKE=debian 2022-03-10T10:10:39.040 INFO:teuthology.orchestra.run.smithi025.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-10T10:10:39.041 INFO:teuthology.orchestra.run.smithi025.stdout:VERSION_ID="20.04" 2022-03-10T10:10:39.041 INFO:teuthology.orchestra.run.smithi025.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-10T10:10:39.042 INFO:teuthology.orchestra.run.smithi025.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-10T10:10:39.042 INFO:teuthology.orchestra.run.smithi025.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-10T10:10:39.042 INFO:teuthology.orchestra.run.smithi025.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-10T10:10:39.043 INFO:teuthology.orchestra.run.smithi025.stdout:VERSION_CODENAME=focal 2022-03-10T10:10:39.043 INFO:teuthology.orchestra.run.smithi025.stdout:UBUNTU_CODENAME=focal 2022-03-10T10:10:39.045 INFO:teuthology.lock.ops:Updating smithi025.front.sepia.ceph.com on lock server 2022-03-10T10:10:39.062 INFO:teuthology.lock.ops:Updating [smithi153.front.sepia.ceph.com]: set os type and version on server 2022-03-10T10:10:39.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2022-03-10T10:10:39.370 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2022-03-10T10:10:39.377 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="Ubuntu" 2022-03-10T10:10:39.377 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-10T10:10:39.378 INFO:teuthology.orchestra.run.smithi153.stdout:ID=ubuntu 2022-03-10T10:10:39.378 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE=debian 2022-03-10T10:10:39.379 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-10T10:10:39.379 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="20.04" 2022-03-10T10:10:39.379 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-10T10:10:39.380 INFO:teuthology.orchestra.run.smithi153.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-10T10:10:39.380 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-10T10:10:39.381 INFO:teuthology.orchestra.run.smithi153.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-10T10:10:39.381 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_CODENAME=focal 2022-03-10T10:10:39.381 INFO:teuthology.orchestra.run.smithi153.stdout:UBUNTU_CODENAME=focal 2022-03-10T10:10:39.382 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2022-03-10T10:10:39.478 INFO:teuthology.dispatcher.supervisor:Running job 6729612 2022-03-10T10:10:39.479 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-03-10_05:07:01-rgw-pacific-distro-default-smithi/6729612 --name teuthology-2022-03-10_05:07:01-rgw-pacific-distro-default-smithi --description rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} -- /home/teuthworker/archive/teuthology-2022-03-10_05:07:01-rgw-pacific-distro-default-smithi/6729612/orig.config.yaml 2022-03-10T10:10:39.482 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-03-10_05:07:01-rgw-pacific-distro-default-smithi/6729612 2022-03-10T10:10:39.483 INFO:teuthology.dispatcher.supervisor:Job PID: 20732 2022-03-10T10:10:39.483 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-03-10T16:40:50.412 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-03-10T16:40:50.579 INFO:teuthology.kill:Killing Pids: {20732} 2022-03-10T16:40:51.751 INFO:teuthology.task.internal:roles: ubuntu@smithi025.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2022-03-10T16:40:51.752 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2022-03-10T16:40:51.753 INFO:teuthology.misc:Compressing logs... 2022-03-10T16:40:51.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi025.front.sepia.ceph.com' 2022-03-10T16:40:51.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2022-03-10T16:40:52.212 INFO:teuthology.misc:Compressing logs... 2022-03-10T16:40:52.310 INFO:teuthology.misc:Compressing logs... 2022-03-10T16:42:57.589 INFO:teuthology.kill:No teuthology processes running 2022-03-10T16:42:58.874 INFO:teuthology.kill:Nuking machines: ['smithi025', 'smithi153'] 2022-03-10T16:42:59.990 INFO:teuthology.kill:2022-03-10 16:42:59,990.990 INFO:teuthology.nuke:targets: 2022-03-10T16:42:59.991 INFO:teuthology.kill: smithi025.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvYDmCyh5S8yKhKZJ28DOiDn6cFnp9/V+RdLoRyzqtROJklL/lt9xL0UNQmXwR8ZWAtiHALegRtDyMyjF7oGA7t78pHzndwD8XXoSR7hIaiJcBOwMqbE2y4OFoX3bIpl85yrnV7xkRflSM3oVeZmWNdBz5bxFIvswkAU0MbbwjxIcCgpgB8I9OJhXC+tBfACOpiw+1ew8X3pwSz0FzDY6AOfRyLExseUSKDK7WnCqDsqT/uU9YB2C+QCvVyrDkJdfnVurf/TbdLOXd0xAhBHENBgmclh1xeY8gwoG9hW7uCFT8Qf5qLfcPd5/DABdakvRH2EUypmrOe+RPjibKbQTk6eEyjIk56v/KX2oyRZzTOjKgfIVynkl9odE/N6zD/L9apDTdxF8ZVNQXXzhs3CfsdUdkUJEYn/X55d4ZrIFMiCLrrY+o5nQHQXuu1pfzYCPcFKdAVMCoMUuUh9+v7dbwJsNaKJ+Kbjd3JbsmAwjO9gI6EEmSyoxyPux3Glwr0J0= 2022-03-10T16:42:59.991 INFO:teuthology.kill: smithi153.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdH1W+GE8h2Pw5e4fNjNvtzHhUtKFsWHBjW+vEYcHJEa0Eit/WnyeiwyNBb/O4r5A7LYZpnnhiym1Vtq8ntFOS4zWdy9Jwb3g2v5MvbM8OlGzsNAxnr9nsbC/gjLHv0TTy9crIbjPNyPzjWWltUDiK5KYGoaC4wQDnZjiCPWxX8ZQMaMk4bzKvXUQOHr5B/VGtG8y0oGCZZm5GtDRic5E/G0PJql9jXzdrDbtlh8wqaIvcrIdJ/UB8kaDnsMVh0Ho57GzhBnX8ZitJnhGpS1W4mBfQZ51n3gM0VDefAeW0lTVLUOjmles09eozf0ZkDBqo+ikOLU6mxOBy0AH+Iunh14T3JPJuwMt4VX0JTZ6f6CSGtlW/7otR71IJU+aTp/nsfK+EsqMdqHo8Zo3YZEEpckoILfN1nTx+a+e1BBDF1sf6MgkhZEsJGqdz8/EWkwE0T3xxosj8I9grERCoHyJVOK42pGfP6dzhJjEBtR3ktwBt09Bbfu4ktJqywRZ5ops= 2022-03-10T16:43:00.017 INFO:teuthology.kill:2022-03-10 16:43:00,017.017 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-10T16:43:00.021 INFO:teuthology.kill:2022-03-10 16:43:00,021.021 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-10T16:43:00.102 INFO:teuthology.kill:2022-03-10 16:43:00,102.102 INFO:teuthology.orchestra.console:Power off smithi025 2022-03-10T16:43:00.117 INFO:teuthology.kill:2022-03-10 16:43:00,117.117 INFO:teuthology.orchestra.console:Power off smithi153 2022-03-10T16:43:12.601 INFO:teuthology.kill:2022-03-10 16:43:12,600.600 INFO:teuthology.orchestra.console:Power off for smithi025 completed 2022-03-10T16:43:12.607 INFO:teuthology.kill:2022-03-10 16:43:12,607.607 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2022-03-10T16:43:12.752 INFO:teuthology.kill:2022-03-10 16:43:12,752.752 INFO:teuthology.lock.ops:unlocked: smithi153.front.sepia.ceph.com 2022-03-10T16:43:12.755 INFO:teuthology.kill:2022-03-10 16:43:12,754.754 INFO:teuthology.lock.ops:unlocked: smithi025.front.sepia.ceph.com 2022-03-10T16:45:12.954 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-03-10T16:45:13.052 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi025 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-03-10_15:28:20-rados-wip-yuri11-testing-2022-03-09-1706-octopus-distro-default-smithi/6730393 2022-03-10T16:45:13.052 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi153 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-03-10_15:28:20-rados-wip-yuri11-testing-2022-03-09-1706-octopus-distro-default-smithi/6730393