2023-08-31T14:38:02.649 INFO:root:teuthology version: 0.0.1.dev192+gf9a8556 2023-08-31T14:38:02.650 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi} --verbose --archive /home/teuthworker/archive/ivancich-2023-08-31_14:27:53-rgw-wip-eric-testing-1-distro-default-smithi/7385474 --owner scheduled_ivancich@teuthology --name ivancich-2023-08-31_14:27:53-rgw-wip-eric-testing-1-distro-default-smithi -- /home/teuthworker/archive/ivancich-2023-08-31_14:27:53-rgw-wip-eric-testing-1-distro-default-smithi/7385474/orig.config.yaml 2023-08-31T14:38:02.687 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-31T14:38:02.767 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/ivancich-2023-08-31_14:27:53-rgw-wip-eric-testing-1-distro-default-smithi/7385474 branch: wip-eric-testing-1 description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi} email: ivancich@redhat.com first_in_suite: false job_id: '7385474' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: ivancich-2023-08-31_14:27:53-rgw-wip-eric-testing-1-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-eric-testing-1 ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: 5535a92bd618b7e580a2fce32e9dfab6620d1086 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 5535a92bd618b7e580a2fce32e9dfab6620d1086 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 rgw-multisite-tests: args: - tests.py workunit: branch: wip-eric-testing-1 sha1: 5535a92bd618b7e580a2fce32e9dfab6620d1086 owner: scheduled_ivancich@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 831 sha1: 5535a92bd618b7e580a2fce32e9dfab6620d1086 sleep_before_teardown: 0 subset: 1/50 suite: rgw suite_branch: wip-eric-testing-1 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5535a92bd618b7e580a2fce32e9dfab6620d1086/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 5535a92bd618b7e580a2fce32e9dfab6620d1086 targets: smithi159.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCPmmKb6S1fl1LlIBuqeZKqdS1oiKCcO48ISPQMQw/nu37OlvEpvhCKK5eW65zKX70RSuDsLpLDzgXY/b4qL9TS+KkB9Gd25XOBqwDwtCvw7Vbm5U2FQU2d4AQkMN90t68XpgyAL1CytFJJ2vMrlyhzqdfpMQjD8bL+/I1teeVBGSmG5hpA9oT9Y/yw+nL75VJs2l5h77Vx6BhhB8lv3ag0TXxURykzVFp+GJnT1VV6Lc51gTnt2lgvUeFlRBV/8hxPIiZQpVLVeVoMbVAeSI9MeEYwwPXscg0Ch5UCJAgA+kZIIV2Yh8VN2b/ZSxFIcOZywGsAWu7/7b2aaGLJKrseiMH58wEuso77F3g8y4p/21qpnsAHYoc9eKnCEGxp4J3Jp9cbk1Kad9qFTHrU/mvidOsZmPDvfP4rvy7nNCKtoqr8dWE5GeqAbsy3jekzzgoWbkrnxOmZXxIPVDlSgI+OfW0sBDLqZRQvu8+l2BnYPsLVkxx1VeHhXFi7c8ANaAU= smithi183.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqg18aL1ck2dcSPO0B7VnNScQjvMOjNqdUKGEFV0ZhwQouD1lRvhVSHAS+R6hmCWmuiPexVC3gZpgQh5mE1AYlULivMHJxPb9xkX5hWsJdPOUl3PNWCkkZP6oaum+zqH5/PXdlHq90yaWdDh32Ciha7GrpUWiE6P5NJxtVZpuxQ6VvQA3LgU+h/9rW9besMZLpcCrteQ8QWCDhiTjut90QG5ng8gleYM0GIQGRyimxOcefGijhe4/NL1dllH+i/ur1wRv15TwY0TtMvSYOCuU3X8CDl3/KaLjmxqBPzxjIjvYqyS7pM2Qutqzts2P82256wcrodi7s5dMqCBQaKcPYwEkpQYPKhuXkURcBXx3tV1fFCkAp61tN4W7+1Nfvq9po6SB5nOQu6vXdmHlxzbmIs3PS3AV87VNZVuzkTWNUjMsPNfiZp4fGajsSDj8PYxkVOWHYY2Dk5XXIch3Kv8bnkODYaSrNb3WqjTKoeaVSvktNSmflXYH4GCDK06Te3o8= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f9a8556d5e989d27b6626a0b75ef595991f83d20 timestamp: 2023-08-31_14:27:53 tube: smithi user: ivancich verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1394 2023-08-31T14:38:02.767 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5535a92bd618b7e580a2fce32e9dfab6620d1086/qa; will attempt to use it 2023-08-31T14:38:02.768 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5535a92bd618b7e580a2fce32e9dfab6620d1086/qa/tasks 2023-08-31T14:38:02.768 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-08-31T14:38:02.770 INFO:teuthology.task.internal:Checking packages... 2023-08-31T14:38:02.790 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5535a92bd618b7e580a2fce32e9dfab6620d1086' 2023-08-31T14:38:02.791 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-08-31T14:38:02.791 INFO:teuthology.packaging:ref: None 2023-08-31T14:38:02.791 INFO:teuthology.packaging:tag: None 2023-08-31T14:38:02.792 INFO:teuthology.packaging:branch: wip-eric-testing-1 2023-08-31T14:38:02.792 INFO:teuthology.packaging:sha1: 5535a92bd618b7e580a2fce32e9dfab6620d1086 2023-08-31T14:38:02.792 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-eric-testing-1 2023-08-31T14:38:02.928 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5801.g5535a92b 2023-08-31T14:38:02.929 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-08-31T14:38:02.934 INFO:teuthology.task.internal:no buildpackages task found 2023-08-31T14:38:02.935 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-08-31T14:38:02.947 INFO:teuthology.task.internal:Saving configuration 2023-08-31T14:38:02.961 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-08-31T14:38:02.966 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-31T14:38:02.989 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ivancich-2023-08-31_14:27:53-rgw-wip-eric-testing-1-distro-default-smithi/7385474', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 14:28:58.113997', 'locked_by': 'scheduled_ivancich@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBklL90gdL3/swKcyWp/QJb5bp9B3etaMKLrz6NXRyblrlOyWzRlCj1CaV92lnLhDGt0/qZcdWQMLGkYKH9QIpw='} 2023-08-31T14:38:03.013 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ivancich-2023-08-31_14:27:53-rgw-wip-eric-testing-1-distro-default-smithi/7385474', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 14:28:58.115531', 'locked_by': 'scheduled_ivancich@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKj9u5j1DCasPiJSXnqzQB83NlUW/ystjLGhI1JQuqf0kwPiTUSJxT0oyo8itDtAom7uBMteuK1wBK3ba0DyvYg='} 2023-08-31T14:38:03.013 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-08-31T14:38:03.020 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-08-31T14:38:03.021 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-08-31T14:38:03.021 INFO:teuthology.run_tasks:Running task console_log... 2023-08-31T14:38:03.083 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f71359b4af0>, signals=[15]) 2023-08-31T14:38:03.084 INFO:teuthology.run_tasks:Running task internal.connect... 2023-08-31T14:38:03.090 INFO:teuthology.task.internal:Opening connections... 2023-08-31T14:38:03.091 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi159.front.sepia.ceph.com 2023-08-31T14:38:03.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-31T14:38:03.110 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/task/internal/__init__.py", line 180, in connect rem.connect() File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/orchestra/remote.py", line 386, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 421, in connect t.start_client(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 699, in start_client raise e File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 303, in read_all raise EOFError() EOFError 2023-08-31T14:38:03.169 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4f5db49da50041f396d9b524bb332952 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/task/internal/__init__.py", line 180, in connect rem.connect() File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/orchestra/remote.py", line 386, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 421, in connect t.start_client(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 699, in start_client raise e File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_f9a8556d5e989d27b6626a0b75ef595991f83d20/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 303, in read_all raise EOFError() EOFError 2023-08-31T14:38:03.173 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-08-31T14:38:03.185 INFO:teuthology.nuke:Checking targets against current locks 2023-08-31T14:38:03.223 DEBUG:teuthology.nuke:shortname: smithi159 2023-08-31T14:38:03.224 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-31T14:38:03.245 DEBUG:teuthology.nuke:shortname: smithi183 2023-08-31T14:38:03.246 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-31T14:38:03.249 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ivancich-2023-08-31_14:27:53-rgw-wip-eric-testing-1-distro-default-smithi/7385474', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 14:28:58.113997', 'locked_by': 'scheduled_ivancich@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBklL90gdL3/swKcyWp/QJb5bp9B3etaMKLrz6NXRyblrlOyWzRlCj1CaV92lnLhDGt0/qZcdWQMLGkYKH9QIpw='} 2023-08-31T14:38:03.268 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ivancich-2023-08-31_14:27:53-rgw-wip-eric-testing-1-distro-default-smithi/7385474', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 14:28:58.115531', 'locked_by': 'scheduled_ivancich@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKj9u5j1DCasPiJSXnqzQB83NlUW/ystjLGhI1JQuqf0kwPiTUSJxT0oyo8itDtAom7uBMteuK1wBK3ba0DyvYg='} 2023-08-31T14:38:03.299 INFO:teuthology.orchestra.console.smithi159:Power off 2023-08-31T14:38:03.300 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-31T14:38:03.322 DEBUG:teuthology.orchestra.console.smithi159:power off output: Chassis Power Control: Down/Off 2023-08-31T14:38:03.322 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T14:38:03.335 INFO:teuthology.orchestra.console.smithi183:Power off 2023-08-31T14:38:03.336 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-31T14:38:03.344 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2023-08-31T14:38:03.357 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2023-08-31T14:38:03.357 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T14:38:03.378 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2023-08-31T14:38:07.349 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T14:38:07.380 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T14:38:07.458 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2023-08-31T14:38:07.487 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2023-08-31T14:38:11.461 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T14:38:11.488 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T14:38:11.571 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off 2023-08-31T14:38:11.597 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2023-08-31T14:38:11.673 INFO:teuthology.orchestra.console.smithi159:Power off completed 2023-08-31T14:38:15.602 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T14:38:15.711 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2023-08-31T14:38:15.812 INFO:teuthology.orchestra.console.smithi183:Power off completed 2023-08-31T14:38:15.916 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi} failure_reason: '' owner: scheduled_ivancich@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4f5db49da50041f396d9b524bb332952 status: fail success: false 2023-08-31T14:38:15.917 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-31T14:38:15.998 INFO:teuthology.run:FAIL