2021-12-09T10:54:55.614 INFO:root:teuthology version: 1.1.0-95a7d479
2021-12-09T10:54:55.737 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi001.front.sepia.ceph.com' reimaging
2021-12-09T10:54:55.739 INFO:teuthology.lock.ops:Updating [ubuntu@smithi001.front.sepia.ceph.com]: reset os type and version on server
2021-12-09T10:54:55.739 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server
2021-12-09T10:54:55.756 INFO:teuthology.lock.ops:Node 'ubuntu@smithi001.front.sepia.ceph.com' reimaging is complete
2021-12-09T10:54:55.757 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi110.front.sepia.ceph.com' reimaging
2021-12-09T10:54:55.757 INFO:teuthology.lock.ops:Updating [ubuntu@smithi110.front.sepia.ceph.com]: reset os type and version on server
2021-12-09T10:54:55.758 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server
2021-12-09T10:54:55.776 INFO:teuthology.lock.ops:Node 'ubuntu@smithi110.front.sepia.ceph.com' reimaging is complete
2021-12-09T10:54:55.962 INFO:teuthology.provision.fog.smithi001:Scheduling deploy of rhel 8.4
2021-12-09T10:54:55.969 INFO:teuthology.provision.fog.smithi110:Scheduling deploy of rhel 8.4
2021-12-09T10:54:56.267 INFO:teuthology.orchestra.console:Power off smithi110
2021-12-09T10:54:56.282 INFO:teuthology.orchestra.console:Power off smithi001
2021-12-09T10:55:04.636 INFO:teuthology.orchestra.console:Power off for smithi110 completed
2021-12-09T10:55:04.648 INFO:teuthology.orchestra.console:Power off for smithi001 completed
2021-12-09T10:55:04.738 INFO:teuthology.orchestra.console:Power on smithi110
2021-12-09T10:55:04.749 INFO:teuthology.orchestra.console:Power on smithi001
2021-12-09T10:55:13.111 INFO:teuthology.orchestra.console:Power on for smithi110 completed
2021-12-09T10:55:13.116 INFO:teuthology.orchestra.console:Power on for smithi001 completed
2021-12-09T10:55:13.213 INFO:teuthology.provision.fog.smithi110:Waiting for deploy to finish
2021-12-09T10:55:13.217 INFO:teuthology.provision.fog.smithi001:Waiting for deploy to finish
2021-12-09T10:59:44.756 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-12-09T10:59:44.965 INFO:teuthology.provision.fog.smithi001:Node is ready
2021-12-09T10:59:44.981 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001.front.sepia.ceph.com
2021-12-09T10:59:45.038 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1 smithi001.front.sepia.ceph.com smithi001
2021-12-09T10:59:45.364 INFO:teuthology.provision.fog.smithi001:Deploy complete!
2021-12-09T11:00:02.713 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-12-09T11:00:02.923 INFO:teuthology.provision.fog.smithi110:Node is ready
2021-12-09T11:00:02.938 INFO:teuthology.orchestra.run.smithi110.stdout:smithi110.front.sepia.ceph.com
2021-12-09T11:00:02.994 INFO:teuthology.orchestra.run.smithi110.stdout:172.21.15.110 smithi110.front.sepia.ceph.com smithi110
2021-12-09T11:00:03.303 INFO:teuthology.provision.fog.smithi110:Deploy complete!
2021-12-09T11:00:03.451 INFO:teuthology.lock.ops:Checking smithi001.front.sepia.ceph.com
2021-12-09T11:00:03.452 INFO:teuthology.lock.ops:New key found. Updating...
2021-12-09T11:00:03.471 INFO:teuthology.lock.ops:Checking smithi110.front.sepia.ceph.com
2021-12-09T11:00:03.471 INFO:teuthology.lock.ops:New key found. Updating...
2021-12-09T11:00:03.489 INFO:teuthology.lock.ops:Updating [smithi001.front.sepia.ceph.com]: set os type and version on server
2021-12-09T11:00:03.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com'
2021-12-09T11:00:03.728 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64
2021-12-09T11:00:03.785 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="Red Hat Enterprise Linux"
2021-12-09T11:00:03.785 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="8.4 (Ootpa)"
2021-12-09T11:00:03.785 INFO:teuthology.orchestra.run.smithi001.stdout:ID="rhel"
2021-12-09T11:00:03.786 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="fedora"
2021-12-09T11:00:03.786 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="8.4"
2021-12-09T11:00:03.786 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el8"
2021-12-09T11:00:03.786 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2021-12-09T11:00:03.787 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31"
2021-12-09T11:00:03.787 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2021-12-09T11:00:03.788 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://www.redhat.com/"
2021-12-09T11:00:03.789 INFO:teuthology.orchestra.run.smithi001.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2021-12-09T11:00:03.789 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-12-09T11:00:03.789 INFO:teuthology.orchestra.run.smithi001.stdout:
2021-12-09T11:00:03.789 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2021-12-09T11:00:03.790 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2021-12-09T11:00:03.790 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2021-12-09T11:00:03.790 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2021-12-09T11:00:03.791 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server
2021-12-09T11:00:03.809 INFO:teuthology.lock.ops:Updating [smithi110.front.sepia.ceph.com]: set os type and version on server
2021-12-09T11:00:03.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com'
2021-12-09T11:00:03.990 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64
2021-12-09T11:00:04.045 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Red Hat Enterprise Linux"
2021-12-09T11:00:04.045 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="8.4 (Ootpa)"
2021-12-09T11:00:04.045 INFO:teuthology.orchestra.run.smithi110.stdout:ID="rhel"
2021-12-09T11:00:04.045 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="fedora"
2021-12-09T11:00:04.046 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="8.4"
2021-12-09T11:00:04.046 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el8"
2021-12-09T11:00:04.046 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2021-12-09T11:00:04.046 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31"
2021-12-09T11:00:04.047 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2021-12-09T11:00:04.047 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.redhat.com/"
2021-12-09T11:00:04.047 INFO:teuthology.orchestra.run.smithi110.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2021-12-09T11:00:04.048 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-12-09T11:00:04.048 INFO:teuthology.orchestra.run.smithi110.stdout:
2021-12-09T11:00:04.048 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2021-12-09T11:00:04.048 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2021-12-09T11:00:04.049 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2021-12-09T11:00:04.049 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2021-12-09T11:00:04.049 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server
2021-12-09T11:00:04.167 INFO:teuthology.dispatcher.supervisor:Running job 6554595
2021-12-09T11:00:04.168 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology -v --owner scheduled_soumyakoduri@teuthology --archive /home/teuthworker/archive/soumyakoduri-2021-12-09_10:46:12-rgw-wip-skoduri-dbstore-mp-distro-basic-smithi/6554595 --name soumyakoduri-2021-12-09_10:46:12-rgw-wip-skoduri-dbstore-mp-distro-basic-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} -- /home/teuthworker/archive/soumyakoduri-2021-12-09_10:46:12-rgw-wip-skoduri-dbstore-mp-distro-basic-smithi/6554595/orig.config.yaml
2021-12-09T11:00:04.172 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/soumyakoduri-2021-12-09_10:46:12-rgw-wip-skoduri-dbstore-mp-distro-basic-smithi/6554595
2021-12-09T11:00:04.172 INFO:teuthology.dispatcher.supervisor:Job PID: 12458
2021-12-09T11:00:04.173 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-12-09T23:00:24.183 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-12-09T23:00:24.341 INFO:teuthology.kill:Killing Pids: {12458}
2021-12-09T23:00:25.638 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1']
2021-12-09T23:00:25.639 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']
2021-12-09T23:00:25.639 INFO:teuthology.misc:Compressing logs...
2021-12-09T23:00:25.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com'
2021-12-09T23:00:25.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com'
2021-12-09T23:00:26.454 INFO:teuthology.misc:Compressing logs...
2021-12-09T23:00:26.576 INFO:teuthology.misc:Compressing logs...
2021-12-09T23:18:14.169 INFO:teuthology.kill:No teuthology processes running
2021-12-09T23:18:15.369 INFO:teuthology.kill:Nuking machines: ['smithi001', 'smithi110']
2021-12-09T23:18:16.329 INFO:teuthology.kill:2021-12-09 23:18:16,329.329 INFO:teuthology.nuke:targets:
2021-12-09T23:18:16.330 INFO:teuthology.kill:  smithi001.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChiWvhvgTqDDzn5LuGJfVWJcZyxAR54DrcENCBztW6jZmNKZ6KEQLrYjSw8WA+pA+vd+wzhwuZ0IKyJsU2UqQKq/YNmDJ5s9jKUzuKogevxmihd954HtMiKSCTcaC16/R95OIZyffH28hVrpmjksSXGQC6JlSzVk6BTg/FuMY0AyPGJnHNS75UaeMP3ZCR11hZwBs6IO5ccqc0BdAkHzEvJmV9sUSYb+TvkUSjgk3OicCFSIvAqNHuRi0CsIr7Bvk1tjr/w6yvLL6NXeDNHzCmtg08ZrQb8+UA1JNqZqPHXfZcA0QQVEMm5X/OHIl6aW1PpZxWfyvxUqe3PyFZl4n0faIvA5jKZJ6QXL1CBXaWgR1PI0eENGD6LdQDoSAgsRUEXiPZpcjrpXIXIK8r10enGPxZroZuJFDmwghW3MzX82niifczpyZnM3UDzsPEBZfym8lLnFhRiyujXrDU6jjlm87C4EX8a+x2vLUaqH351Iz4ZSw1EUgmn7YFJTmwXss=
2021-12-09T23:18:16.331 INFO:teuthology.kill:  smithi110.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzWh0lNIWwEyzc473embBe3I8230TG4wPvfMRzFPLcrVNHoyJpBFKVTRJsbqdD+o4+dKFidD9rOjY9/ZKD9JDPEEoYxSqlkJ7PTtNImeG32VuhoF4RZ1nYoFNIdvDoAVwOXp2Uz/WQgqz7vIDEsQZF0JBmRgtzDAWGcbzYYFfpxZhrT9pjIYQNhkXh4j0FDzF1bUjaRz3yQRSRWC8HknuUqZunzYeRJrP0tJ58OjBKPFmOn4ByBR2WjVI2J9+JVli4I4SFrEtevotY9pZ6r6WolAUhHXDZ+pekw/utRRi232THQ//qE1Fd2sUHKuQZvff0n2LFYhnj1bGIP5RfwiqKgcmu5w0KNsiLQY9nVipimjsEUGJCYrAtFYNdJfbgTh88e+BZ9dg2JBnp7HG+hj++vKBLlNJiWqEdUS3eRn3D3H+6Cil9blEXg6NNaB1vcfyS/Qq97l3MNRHAmO9g/gVkYASCi5pqToE7o3X1dKFynoO6qcRr3DIq8fF0t61jerM=
2021-12-09T23:18:16.357 INFO:teuthology.kill:2021-12-09 23:18:16,357.357 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-12-09T23:18:16.361 INFO:teuthology.kill:2021-12-09 23:18:16,361.361 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-12-09T23:18:16.431 INFO:teuthology.kill:2021-12-09 23:18:16,430.430 INFO:teuthology.orchestra.console:Power off smithi001
2021-12-09T23:18:16.450 INFO:teuthology.kill:2021-12-09 23:18:16,450.450 INFO:teuthology.orchestra.console:Power off smithi110
2021-12-09T23:18:24.811 INFO:teuthology.kill:2021-12-09 23:18:24,810.810 INFO:teuthology.orchestra.console:Power off for smithi001 completed
2021-12-09T23:18:24.815 INFO:teuthology.kill:2021-12-09 23:18:24,815.815 INFO:teuthology.orchestra.console:Power off for smithi110 completed
2021-12-09T23:18:24.963 INFO:teuthology.kill:2021-12-09 23:18:24,963.963 INFO:teuthology.lock.ops:unlocked: smithi110.front.sepia.ceph.com
2021-12-09T23:18:24.968 INFO:teuthology.kill:2021-12-09 23:18:24,968.968 INFO:teuthology.lock.ops:unlocked: smithi001.front.sepia.ceph.com
2021-12-09T23:20:25.263 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-12-09T23:20:25.377 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi001 but it was locked by another job: /home/teuthworker/archive/benhanokh-2021-12-09_12:11:04-rados-WIP_GBH_safe_fast_shutdown_7-distro-basic-smithi/6554964
2021-12-09T23:20:25.378 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi110 but it was locked by another job: /home/teuthworker/archive/benhanokh-2021-12-09_12:11:04-rados-WIP_GBH_safe_fast_shutdown_7-distro-basic-smithi/6554964