2022-03-31T06:33:11.151 INFO:root:teuthology version: 1.1.0-d877b023 2022-03-31T06:33:11.279 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi049.front.sepia.ceph.com' reimaging 2022-03-31T06:33:11.280 INFO:teuthology.lock.ops:Updating [ubuntu@smithi049.front.sepia.ceph.com]: reset os type and version on server 2022-03-31T06:33:11.280 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2022-03-31T06:33:11.299 INFO:teuthology.lock.ops:Node 'ubuntu@smithi049.front.sepia.ceph.com' reimaging is complete 2022-03-31T06:33:11.300 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi103.front.sepia.ceph.com' reimaging 2022-03-31T06:33:11.300 INFO:teuthology.lock.ops:Updating [ubuntu@smithi103.front.sepia.ceph.com]: reset os type and version on server 2022-03-31T06:33:11.300 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2022-03-31T06:33:11.318 INFO:teuthology.lock.ops:Node 'ubuntu@smithi103.front.sepia.ceph.com' reimaging is complete 2022-03-31T06:33:11.512 INFO:teuthology.provision.fog.smithi049:Scheduling deploy of ubuntu 20.04 2022-03-31T06:33:11.522 INFO:teuthology.provision.fog.smithi103:Scheduling deploy of ubuntu 20.04 2022-03-31T06:33:11.817 INFO:teuthology.orchestra.console:Power off smithi103 2022-03-31T06:33:11.856 INFO:teuthology.orchestra.console:Power off smithi049 2022-03-31T06:33:20.218 INFO:teuthology.orchestra.console:Power off for smithi049 completed 2022-03-31T06:33:20.319 INFO:teuthology.orchestra.console:Power on smithi049 2022-03-31T06:33:24.305 INFO:teuthology.orchestra.console:Power off for smithi103 completed 2022-03-31T06:33:24.407 INFO:teuthology.orchestra.console:Power on smithi103 2022-03-31T06:33:28.681 INFO:teuthology.orchestra.console:Power on for smithi049 completed 2022-03-31T06:33:28.782 INFO:teuthology.provision.fog.smithi049:Waiting for deploy to finish 2022-03-31T06:33:32.772 INFO:teuthology.orchestra.console:Power on for smithi103 completed 2022-03-31T06:33:32.873 INFO:teuthology.provision.fog.smithi103:Waiting for deploy to finish 2022-03-31T06:39:07.224 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-31T06:39:07.612 INFO:teuthology.provision.fog.smithi103:Node is ready 2022-03-31T06:39:07.619 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103.front.sepia.ceph.com 2022-03-31T06:39:07.627 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103 smithi103.front.sepia.ceph.com smithi103 2022-03-31T06:39:07.871 INFO:teuthology.provision.fog.smithi103:Deploy complete! 2022-03-31T06:39:16.489 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-31T06:39:16.982 INFO:teuthology.provision.fog.smithi049:Node is ready 2022-03-31T06:39:16.988 INFO:teuthology.orchestra.run.smithi049.stdout:smithi049.front.sepia.ceph.com 2022-03-31T06:39:16.996 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.15.49 smithi049.front.sepia.ceph.com smithi049 2022-03-31T06:39:17.223 INFO:teuthology.provision.fog.smithi049:Deploy complete! 2022-03-31T06:39:17.414 INFO:teuthology.lock.ops:Checking smithi049.front.sepia.ceph.com 2022-03-31T06:39:17.415 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-31T06:39:17.434 INFO:teuthology.lock.ops:Checking smithi103.front.sepia.ceph.com 2022-03-31T06:39:17.434 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-31T06:39:17.454 INFO:teuthology.lock.ops:Updating [smithi049.front.sepia.ceph.com]: set os type and version on server 2022-03-31T06:39:17.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2022-03-31T06:39:17.643 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2022-03-31T06:39:17.691 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="Ubuntu" 2022-03-31T06:39:17.692 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-31T06:39:17.692 INFO:teuthology.orchestra.run.smithi049.stdout:ID=ubuntu 2022-03-31T06:39:17.692 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE=debian 2022-03-31T06:39:17.692 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-31T06:39:17.693 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="20.04" 2022-03-31T06:39:17.693 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-31T06:39:17.693 INFO:teuthology.orchestra.run.smithi049.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-31T06:39:17.694 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-31T06:39:17.694 INFO:teuthology.orchestra.run.smithi049.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-31T06:39:17.694 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_CODENAME=focal 2022-03-31T06:39:17.694 INFO:teuthology.orchestra.run.smithi049.stdout:UBUNTU_CODENAME=focal 2022-03-31T06:39:17.696 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2022-03-31T06:39:17.714 INFO:teuthology.lock.ops:Updating [smithi103.front.sepia.ceph.com]: set os type and version on server 2022-03-31T06:39:17.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2022-03-31T06:39:17.935 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2022-03-31T06:39:17.942 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="Ubuntu" 2022-03-31T06:39:17.942 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-31T06:39:17.942 INFO:teuthology.orchestra.run.smithi103.stdout:ID=ubuntu 2022-03-31T06:39:17.943 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE=debian 2022-03-31T06:39:17.943 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-31T06:39:17.943 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="20.04" 2022-03-31T06:39:17.943 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-31T06:39:17.944 INFO:teuthology.orchestra.run.smithi103.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-31T06:39:17.944 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-31T06:39:17.944 INFO:teuthology.orchestra.run.smithi103.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-31T06:39:17.944 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_CODENAME=focal 2022-03-31T06:39:17.945 INFO:teuthology.orchestra.run.smithi103.stdout:UBUNTU_CODENAME=focal 2022-03-31T06:39:17.945 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2022-03-31T06:39:18.042 INFO:teuthology.dispatcher.supervisor:Running job 6769422 2022-03-31T06:39:18.042 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-03-31_05:07:02-rgw-pacific-distro-default-smithi/6769422 --name teuthology-2022-03-31_05:07:02-rgw-pacific-distro-default-smithi --description rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} -- /home/teuthworker/archive/teuthology-2022-03-31_05:07:02-rgw-pacific-distro-default-smithi/6769422/orig.config.yaml 2022-03-31T06:39:18.046 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-03-31_05:07:02-rgw-pacific-distro-default-smithi/6769422 2022-03-31T06:39:18.047 INFO:teuthology.dispatcher.supervisor:Job PID: 12206 2022-03-31T06:39:18.047 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-03-31T13:09:47.055 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-03-31T13:09:47.303 INFO:teuthology.kill:Killing Pids: {12206} 2022-03-31T13:09:48.521 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2022-03-31T13:09:48.522 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2022-03-31T13:09:48.523 INFO:teuthology.misc:Compressing logs... 2022-03-31T13:09:48.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2022-03-31T13:09:48.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2022-03-31T13:09:49.044 INFO:teuthology.misc:Compressing logs... 2022-03-31T13:09:49.175 INFO:teuthology.misc:Compressing logs... 2022-03-31T13:11:47.448 INFO:teuthology.kill:No teuthology processes running 2022-03-31T13:11:48.675 INFO:teuthology.kill:Nuking machines: ['smithi049', 'smithi103'] 2022-03-31T13:11:49.699 INFO:teuthology.kill:2022-03-31 13:11:49,699.699 INFO:teuthology.nuke:targets: 2022-03-31T13:11:49.700 INFO:teuthology.kill: smithi049.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCTR649Ct9OOtoWudLh/gClr+dJ3RiX1sHwVQ/gu8Ucaz3UywpncT0dBzDgexm9gi8iC46HhoHkRBsQKkYKFwkK0XOR51msdNOUMTx2nxCOfvzLE9wl7pNh7BKjJJ5SA5sDMGjhEO7XHvtb8VaakQ5ctrrfrql8uwy0swXCw67c80yGfbwBlIurPAIG/feawU2493+dq6FPwKVJX5SGTSIxTP9wSm3DvDwxoleUoeUkafsJlBS9bECPFgd+N/1swBHKicRbeO/7AcADGj7Q+rI91x49r2x/XhP0oDPIffDfyLKaxW2nSKMuE2IeHq8wZ3NtHhAe8tdIUefe8b62+QYhsd2Np68jtM63/rOJAEd9qJFsASM0vuJx0famsWyq9DtHxSXiEScqbMDGenOrFgIffqRc4N/meV2DhkJmLHKjyR6DRzDIy4EPEP/2Vz2GUJz+j3DfqIOTs0EcPjWFEb0KicNMpW6bu7XBjTEzmDdY0kUn0696JTqFd1IXeMJWDGk= 2022-03-31T13:11:49.700 INFO:teuthology.kill: smithi103.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2gncimrqYB8ix/2XkrKOTNR8taWa9hyhOnEwtSkPa6xN5WLM+vEqQ1yUJovz8fZNbi/v1fxhv0gFwkHoVDrHUeUPnLwq1HVGegttemcYzS/CmeYvdmL53YHLcWwvrB6X2Xbq7N+wBq3+E9m0Zq4X8OFt97nz1y1JAh+MX3niBHLs9wP1i3hnQ29+SHuBiOXApz0hViXkKR/DMB2s/ErXBpk3lZ/6mujfIjThzxpo3VMm+gz+xWCk+Z7Owlrmpfrv4moEdW9x8kwtHQXyv6kHgqOlrxZ+2W6HXlWTupppKK9VVhljIpiLIfIQ697l7OPpat0z21Q0UyMgV41ovVG9c7GpW/SaCFIC2yil48P/4V+nPqWko3Cor1KeFnP1B6lssCPNgjL12wgM5CfTlX+m78p5YX5Zt/5Z0jfM9b+hG3Ip1uU5aULD/NVsQFjZyFUbdu2u2WdAnE7tPRAUvVOpZh11S2ANs9oy2VQbIR2fH1utEabube0B1IkK2n9s5RrE= 2022-03-31T13:11:49.728 INFO:teuthology.kill:2022-03-31 13:11:49,728.728 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-31T13:11:49.730 INFO:teuthology.kill:2022-03-31 13:11:49,730.730 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-31T13:11:49.811 INFO:teuthology.kill:2022-03-31 13:11:49,810.810 INFO:teuthology.orchestra.console:Power off smithi103 2022-03-31T13:11:49.830 INFO:teuthology.kill:2022-03-31 13:11:49,829.829 INFO:teuthology.orchestra.console:Power off smithi049 2022-03-31T13:12:02.313 INFO:teuthology.kill:2022-03-31 13:12:02,313.313 INFO:teuthology.orchestra.console:Power off for smithi103 completed 2022-03-31T13:12:02.316 INFO:teuthology.kill:2022-03-31 13:12:02,316.316 INFO:teuthology.orchestra.console:Power off for smithi049 completed 2022-03-31T13:12:02.466 INFO:teuthology.kill:2022-03-31 13:12:02,465.465 INFO:teuthology.lock.ops:unlocked: smithi049.front.sepia.ceph.com 2022-03-31T13:12:02.472 INFO:teuthology.kill:2022-03-31 13:12:02,472.472 INFO:teuthology.lock.ops:unlocked: smithi103.front.sepia.ceph.com 2022-03-31T13:14:02.845 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-03-31T13:14:02.955 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi049 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-03-27_12:15:03-powercycle-octopus-distro-default-smithi/6763749 2022-03-31T13:14:02.955 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi103 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-03-27_12:15:03-powercycle-octopus-distro-default-smithi/6763749