2022-07-19T18:38:20.938 INFO:root:teuthology version: 1.1.1.dev383+g89d874d5 2022-07-19T18:38:21.089 INFO:teuthology.lock.ops:Start node 'smithi037.front.sepia.ceph.com' reimaging 2022-07-19T18:38:21.090 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: reset os type and version on server 2022-07-19T18:38:21.090 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2022-07-19T18:38:21.111 INFO:teuthology.lock.ops:Node 'smithi037.front.sepia.ceph.com' reimaging is complete 2022-07-19T18:38:21.111 INFO:teuthology.lock.ops:Start node 'smithi074.front.sepia.ceph.com' reimaging 2022-07-19T18:38:21.111 INFO:teuthology.lock.ops:Updating [smithi074.front.sepia.ceph.com]: reset os type and version on server 2022-07-19T18:38:21.112 INFO:teuthology.lock.ops:Updating smithi074.front.sepia.ceph.com on lock server 2022-07-19T18:38:21.135 INFO:teuthology.lock.ops:Node 'smithi074.front.sepia.ceph.com' reimaging is complete 2022-07-19T18:38:21.331 INFO:teuthology.provision.fog.smithi037:Scheduling deploy of rhel 8.6 2022-07-19T18:38:21.358 INFO:teuthology.provision.fog.smithi074:Scheduling deploy of rhel 8.6 2022-07-19T18:38:21.669 INFO:teuthology.orchestra.console:Power off smithi037 2022-07-19T18:38:21.763 INFO:teuthology.orchestra.console:Power off smithi074 2022-07-19T18:38:30.136 INFO:teuthology.orchestra.console:Power off for smithi074 completed 2022-07-19T18:38:30.237 INFO:teuthology.orchestra.console:Power on smithi074 2022-07-19T18:38:34.158 INFO:teuthology.orchestra.console:Power off for smithi037 completed 2022-07-19T18:38:34.259 INFO:teuthology.orchestra.console:Power on smithi037 2022-07-19T18:38:38.521 INFO:teuthology.orchestra.console:Power on for smithi037 completed 2022-07-19T18:38:38.611 INFO:teuthology.orchestra.console:Power on for smithi074 completed 2022-07-19T18:38:38.623 INFO:teuthology.provision.fog.smithi037:Waiting for deploy to finish 2022-07-19T18:38:38.714 INFO:teuthology.provision.fog.smithi074:Waiting for deploy to finish 2022-07-19T18:42:21.378 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-07-19T18:42:41.510 INFO:teuthology.provision.fog.smithi037:Node is ready 2022-07-19T18:42:41.528 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037.front.sepia.ceph.com 2022-07-19T18:42:41.586 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37 smithi037.front.sepia.ceph.com smithi037 2022-07-19T18:42:41.945 INFO:teuthology.provision.fog.smithi037:Deploy complete! 2022-07-19T18:42:49.214 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-07-19T18:43:14.480 INFO:teuthology.provision.fog.smithi074:Node is ready 2022-07-19T18:43:14.500 INFO:teuthology.orchestra.run.smithi074.stdout:smithi074.front.sepia.ceph.com 2022-07-19T18:43:14.558 INFO:teuthology.orchestra.run.smithi074.stdout:172.21.15.74 smithi074.front.sepia.ceph.com smithi074 2022-07-19T18:43:14.925 INFO:teuthology.provision.fog.smithi074:Deploy complete! 2022-07-19T18:43:15.144 INFO:teuthology.lock.ops:Checking smithi037.front.sepia.ceph.com 2022-07-19T18:43:15.145 INFO:teuthology.lock.ops:New key found. Updating... 2022-07-19T18:43:15.165 INFO:teuthology.lock.ops:Checking smithi074.front.sepia.ceph.com 2022-07-19T18:43:15.166 INFO:teuthology.lock.ops:New key found. Updating... 2022-07-19T18:43:15.185 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: set os type and version on server 2022-07-19T18:43:15.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2022-07-19T18:43:15.458 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2022-07-19T18:43:15.516 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Red Hat Enterprise Linux" 2022-07-19T18:43:15.517 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="8.6 (Ootpa)" 2022-07-19T18:43:15.517 INFO:teuthology.orchestra.run.smithi037.stdout:ID="rhel" 2022-07-19T18:43:15.517 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="fedora" 2022-07-19T18:43:15.517 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="8.6" 2022-07-19T18:43:15.518 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el8" 2022-07-19T18:43:15.518 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-07-19T18:43:15.518 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31" 2022-07-19T18:43:15.518 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-07-19T18:43:15.519 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.redhat.com/" 2022-07-19T18:43:15.519 INFO:teuthology.orchestra.run.smithi037.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-07-19T18:43:15.519 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-07-19T18:43:15.519 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-07-19T18:43:15.520 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-07-19T18:43:15.520 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-07-19T18:43:15.520 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-07-19T18:43:15.520 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-07-19T18:43:15.522 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2022-07-19T18:43:15.543 INFO:teuthology.lock.ops:Updating [smithi074.front.sepia.ceph.com]: set os type and version on server 2022-07-19T18:43:15.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi074.front.sepia.ceph.com' 2022-07-19T18:43:15.741 INFO:teuthology.orchestra.run.smithi074.stdout:x86_64 2022-07-19T18:43:15.800 INFO:teuthology.orchestra.run.smithi074.stdout:NAME="Red Hat Enterprise Linux" 2022-07-19T18:43:15.800 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION="8.6 (Ootpa)" 2022-07-19T18:43:15.801 INFO:teuthology.orchestra.run.smithi074.stdout:ID="rhel" 2022-07-19T18:43:15.801 INFO:teuthology.orchestra.run.smithi074.stdout:ID_LIKE="fedora" 2022-07-19T18:43:15.802 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION_ID="8.6" 2022-07-19T18:43:15.802 INFO:teuthology.orchestra.run.smithi074.stdout:PLATFORM_ID="platform:el8" 2022-07-19T18:43:15.803 INFO:teuthology.orchestra.run.smithi074.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-07-19T18:43:15.803 INFO:teuthology.orchestra.run.smithi074.stdout:ANSI_COLOR="0;31" 2022-07-19T18:43:15.804 INFO:teuthology.orchestra.run.smithi074.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-07-19T18:43:15.804 INFO:teuthology.orchestra.run.smithi074.stdout:HOME_URL="https://www.redhat.com/" 2022-07-19T18:43:15.804 INFO:teuthology.orchestra.run.smithi074.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-07-19T18:43:15.805 INFO:teuthology.orchestra.run.smithi074.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-07-19T18:43:15.806 INFO:teuthology.orchestra.run.smithi074.stdout: 2022-07-19T18:43:15.806 INFO:teuthology.orchestra.run.smithi074.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-07-19T18:43:15.807 INFO:teuthology.orchestra.run.smithi074.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-07-19T18:43:15.807 INFO:teuthology.orchestra.run.smithi074.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-07-19T18:43:15.808 INFO:teuthology.orchestra.run.smithi074.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-07-19T18:43:15.809 INFO:teuthology.lock.ops:Updating smithi074.front.sepia.ceph.com on lock server 2022-07-19T18:43:15.921 INFO:teuthology.dispatcher.supervisor:Running job 6938400 2022-07-19T18:43:15.922 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/bin/teuthology -v --owner scheduled_amaredia@teuthology --archive /home/teuthworker/archive/amaredia-2022-07-19_18:37:19-rgw-main-distro-default-smithi/6938400 --name amaredia-2022-07-19_18:37:19-rgw-main-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} -- /home/teuthworker/archive/amaredia-2022-07-19_18:37:19-rgw-main-distro-default-smithi/6938400/orig.config.yaml 2022-07-19T18:43:15.926 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/amaredia-2022-07-19_18:37:19-rgw-main-distro-default-smithi/6938400 2022-07-19T18:43:15.927 INFO:teuthology.dispatcher.supervisor:Job PID: 31036 2022-07-19T18:43:15.927 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-07-20T06:44:36.343 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-07-20T06:44:36.627 INFO:teuthology.kill:Killing Pids: {31036} 2022-07-20T06:44:38.883 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2022-07-20T06:44:38.884 INFO:teuthology.task.internal:roles: ubuntu@smithi074.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2022-07-20T06:44:38.885 INFO:teuthology.misc:Compressing logs... 2022-07-20T06:44:38.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2022-07-20T06:44:39.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi074.front.sepia.ceph.com' 2022-07-20T06:44:39.564 INFO:teuthology.misc:Compressing logs... 2022-07-20T06:44:39.701 INFO:teuthology.misc:Compressing logs... 2022-07-20T06:54:28.414 INFO:teuthology.kill:No teuthology processes running 2022-07-20T06:54:29.828 INFO:teuthology.kill:Nuking machines: ['smithi037', 'smithi074'] 2022-07-20T06:54:31.091 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-07-20T06:54:31.092 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-07-20T06:54:31.092 INFO:teuthology.kill:2022-07-20 06:54:31,091.091 INFO:teuthology.nuke:targets: 2022-07-20T06:54:31.092 INFO:teuthology.kill: smithi037.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD5fszzHjtKMoREK73UA+JmL80RXbZwOVHZKXCTFXpXCvsRsobjCgBoXSXVjS/5zdGg8D4ezQKXf+LMJM5w5EgxR1nbjn5Y0m0P8aqtQ2zpbISFSGSO5hHssHAflzoMScKr+091OEisi0bN1Zwj/qSfNk312nH5FfgZLpFmrT20o/vVC0WIaO4cIMg8FlL9Y6PLPYyTXarzjsgDzFP7cVJUXYrAnjSzx7SpMevev7eFFHI6ipk6zD3Y6oHw/2A9DT0w5Deatt3walWaa0QjxiU2jU14tn4lCbRq/eJBKOIR+g5yzmctEBjCE+TPh4kjpt0ggR4iPQs9wTb7j70VTqlPWHFOrnidinkRqeYy2GPkzpaKTCK5V9wc61YVd+e8nO8mgpsbfRJzLgNedZ6eH+gPODsgMBt5CraJd3v33Fq36stkHr99kDoUYCQtbG6IROF8d8SClSpGH4w7ZQo/PE6Vvk9qVWEs+CtRJAyyO5w7aUv+U46cBEtCph4xYTwu2wE= 2022-07-20T06:54:31.093 INFO:teuthology.kill: smithi074.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRsnF26aIcVm4Npl3yp93UL8g/ajhSkvNAVvWLtVL+WwvH+GzPZxEvCAeyOnPqLO/lW3gucTipVfwRMhKP70kf8wQLDMgpNJvXDeFy2ogme4uD6adqVrB8ke4yMSdoKSxOULFXoF/zsTCZS6Y1ZwQBizuDhnevBPU8yU0l4OfQhyUP5UUV8sFAZZH+6a+ZFAg6enqvvtXkqREu8R0Mb/4FZ9OQvYPI1T3KnWDj3YV09s11pi/TyrXJDh+jxLIFRcS3u4MUJ/RfD06CjqghJS90cJHDs2KlG/QmEMzpXiTfFpHRV80uCB3vJaebTDenTdRkR+KifalhNEi8bTDDYa99OoAvhmrc+vUv6qD9kZW7O2TMsU9pxHtpeSQNP+4i+CP9+GEBwNfyKLgxJXVV6KZtA4VNu6AraK9u7RUSoO2qg8u50Z4/BzR16eIuWPHbR9WaQyl7/nlf0oxjuXbU3UEIi6I3mZM76tp7LAzT5dAeJyZHHBx5gXGfIMQ4mvg1/wc= 2022-07-20T06:54:31.093 INFO:teuthology.kill:2022-07-20 06:54:31,093.093 INFO:teuthology.nuke:Checking targets against current locks 2022-07-20T06:54:31.170 INFO:teuthology.kill:2022-07-20 06:54:31,170.170 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-20T06:54:31.198 INFO:teuthology.kill:2022-07-20 06:54:31,198.198 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-20T06:54:31.283 INFO:teuthology.kill:2022-07-20 06:54:31,283.283 INFO:teuthology.orchestra.console:Power off smithi037 2022-07-20T06:54:31.311 INFO:teuthology.kill:2022-07-20 06:54:31,311.311 INFO:teuthology.orchestra.console:Power off smithi074 2022-07-20T06:54:39.671 INFO:teuthology.kill:2022-07-20 06:54:39,671.671 INFO:teuthology.orchestra.console:Power off for smithi037 completed 2022-07-20T06:54:39.847 INFO:teuthology.kill:2022-07-20 06:54:39,845.845 INFO:teuthology.lock.ops:unlocked: smithi037.front.sepia.ceph.com 2022-07-20T06:54:43.798 INFO:teuthology.kill:2022-07-20 06:54:43,797.797 INFO:teuthology.orchestra.console:Power off for smithi074 completed 2022-07-20T06:54:43.949 INFO:teuthology.kill:2022-07-20 06:54:43,948.948 INFO:teuthology.lock.ops:unlocked: smithi074.front.sepia.ceph.com 2022-07-20T06:56:44.336 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-07-20T06:56:44.520 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi037 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-07-19_23:25:12-rados-wip-yuri2-testing-2022-07-15-0755-pacific-distro-default-smithi/6939329 2022-07-20T06:56:44.521 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi074 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-07-19_23:25:12-rados-wip-yuri2-testing-2022-07-15-0755-pacific-distro-default-smithi/6939331