2023-12-19T20:05:10.885 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-19T20:05:10.885 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} --archive /home/teuthworker/archive/smanjara-2023-12-19_19:59:19-rgw:multisite-wip-shilpa-rgw-test-multisite-distro-default-smithi/7497354 --verbose --name smanjara-2023-12-19_19:59:19-rgw:multisite-wip-shilpa-rgw-test-multisite-distro-default-smithi --owner scheduled_smanjara@teuthology -- /home/teuthworker/archive/smanjara-2023-12-19_19:59:19-rgw:multisite-wip-shilpa-rgw-test-multisite-distro-default-smithi/7497354/orig.config.yaml 2023-12-19T20:05:10.922 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-19T20:05:11.005 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/smanjara-2023-12-19_19:59:19-rgw:multisite-wip-shilpa-rgw-test-multisite-distro-default-smithi/7497354 branch: wip-shilpa-rgw-test-multisite description: rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} email: smanjara@redhat.com first_in_suite: false job_id: '7497354' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: smanjara-2023-12-19_19:59:19-rgw:multisite-wip-shilpa-rgw-test-multisite-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-shilpa-rgw-test-multisite ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw data sync poll interval: 5 rgw md log max shards: 4 rgw meta sync poll interval: 5 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true rgw_thread_pool_size: 2048 setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: c8caac514df88aea80b36c1fb8e351c80720a337 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: c8caac514df88aea80b36c1fb8e351c80720a337 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 rgw-multisite-tests: args: - tests.py workunit: branch: wip-shilpa-rgw-test-multisite sha1: c8caac514df88aea80b36c1fb8e351c80720a337 owner: scheduled_smanjara@teuthology priority: 49 repo: https://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - - c1.mgr.x - c1.osd.3 - c1.osd.4 - c1.osd.5 - c1.client.1 - - c2.mon.a - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - - c2.mgr.x - c2.osd.3 - c2.osd.4 - c2.osd.5 - c2.client.1 sha1: c8caac514df88aea80b36c1fb8e351c80720a337 sleep_before_teardown: 0 suite: rgw:multisite suite_branch: wip-shilpa-rgw-test-multisite suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c8caac514df88aea80b36c1fb8e351c80720a337/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c8caac514df88aea80b36c1fb8e351c80720a337 targets: smithi008.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDrFT7cLQAieJdZExkrKAmViTWtKMd3zb09rcKYy0dlkWhwm+X8JlvJM+s5XdOsRCs3mF4sA/5iyo5yuapu1SEU= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMs2nmqjjHgLryX1Hskb1OxH8Pmy/pDJ0g+Ja244iH4voLYznSxtu2MhDAGg7V2ryA1Q/HIxNdnjz0Fk5BZAhiA= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH81AtGnYEarGoyt8yySdPuQ2fh3SNAb5n8BGiRglCfodTFDEt+mn4LTf2kgm4W8xRqcZ3gzjZkHtAyU4bcJwgM= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGto0wgREZT7IKRt5TM8u5sT7ARHOHVhWqZg5TQ2MvUmygnqFgOG7YKxQS8XVxnmC79zL9Lh4r0Jzrrrh/lynoo= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-19_19:59:19 tube: smithi user: smanjara verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-19T20:05:11.005 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c8caac514df88aea80b36c1fb8e351c80720a337/qa; will attempt to use it 2023-12-19T20:05:11.006 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c8caac514df88aea80b36c1fb8e351c80720a337/qa/tasks 2023-12-19T20:05:11.006 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-19T20:05:11.007 INFO:teuthology.task.internal:Checking packages... 2023-12-19T20:05:11.031 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'c8caac514df88aea80b36c1fb8e351c80720a337' 2023-12-19T20:05:11.031 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T20:05:11.031 INFO:teuthology.packaging:ref: None 2023-12-19T20:05:11.031 INFO:teuthology.packaging:tag: None 2023-12-19T20:05:11.031 INFO:teuthology.packaging:branch: wip-shilpa-rgw-test-multisite 2023-12-19T20:05:11.031 INFO:teuthology.packaging:sha1: c8caac514df88aea80b36c1fb8e351c80720a337 2023-12-19T20:05:11.031 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-shilpa-rgw-test-multisite 2023-12-19T20:05:11.268 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-186-gc8caac51-1jammy 2023-12-19T20:05:11.269 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-19T20:05:11.276 INFO:teuthology.task.internal:no buildpackages task found 2023-12-19T20:05:11.276 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-19T20:05:11.292 INFO:teuthology.task.internal:Saving configuration 2023-12-19T20:05:11.306 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-19T20:05:11.312 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T20:05:11.340 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi008.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/smanjara-2023-12-19_19:59:19-rgw:multisite-wip-shilpa-rgw-test-multisite-distro-default-smithi/7497354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 19:59:32.190682', 'locked_by': 'scheduled_smanjara@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEiEfzHCr85cuAUF1AAG/ZU30GqYqKl0tbNcVtrZamJZ'} 2023-12-19T20:05:11.361 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/smanjara-2023-12-19_19:59:19-rgw:multisite-wip-shilpa-rgw-test-multisite-distro-default-smithi/7497354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 19:59:32.193298', 'locked_by': 'scheduled_smanjara@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBw8hfVMdAEj8QA+9Cm87VZ+99pHqE+dX520nj0h69jV'} 2023-12-19T20:05:11.384 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/prsrivas-2023-12-19_06:08:46-rgw-wip-rgw-d3n-filter-distro-default-smithi/7496296', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': False, 'locked_since': None, 'locked_by': None, 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMTEz/Xj0I8I2XovNvQpMqUnfMjyWa2sEgrMpB4Zr4md'} 2023-12-19T20:05:11.384 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/check_lock.py", line 28, in check_lock assert status['locked'], \ AssertionError: machine smithi174.front.sepia.ceph.com is not locked 2023-12-19T20:05:11.452 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=84b2a33b52284b149e4688c3fc238d6d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/check_lock.py", line 28, in check_lock assert status['locked'], \ AssertionError: machine smithi174.front.sepia.ceph.com is not locked 2023-12-19T20:05:11.459 INFO:teuthology.nuke:Checking targets against current locks 2023-12-19T20:05:11.504 DEBUG:teuthology.nuke:shortname: smithi008 2023-12-19T20:05:11.505 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T20:05:11.527 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi008.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/smanjara-2023-12-19_19:59:19-rgw:multisite-wip-shilpa-rgw-test-multisite-distro-default-smithi/7497354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 19:59:32.190682', 'locked_by': 'scheduled_smanjara@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEiEfzHCr85cuAUF1AAG/ZU30GqYqKl0tbNcVtrZamJZ'} 2023-12-19T20:05:11.532 DEBUG:teuthology.nuke:shortname: smithi170 2023-12-19T20:05:11.532 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T20:05:11.555 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/smanjara-2023-12-19_19:59:19-rgw:multisite-wip-shilpa-rgw-test-multisite-distro-default-smithi/7497354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 19:59:32.193298', 'locked_by': 'scheduled_smanjara@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBw8hfVMdAEj8QA+9Cm87VZ+99pHqE+dX520nj0h69jV'} 2023-12-19T20:05:11.556 INFO:teuthology.nuke:Not nuking smithi174.front.sepia.ceph.com because description doesn't match: smanjara-2023-12-19_19:59:19-rgw:multisite-wip-shilpa-rgw-test-multisite-distro-default-smithi != /home/teuthworker/archive/prsrivas-2023-12-19_06:08:46-rgw-wip-rgw-d3n-filter-distro-default-smithi/7496296 2023-12-19T20:05:11.579 INFO:teuthology.nuke:Not nuking smithi196.front.sepia.ceph.com because description doesn't match: smanjara-2023-12-19_19:59:19-rgw:multisite-wip-shilpa-rgw-test-multisite-distro-default-smithi != /home/teuthworker/archive/prsrivas-2023-12-19_06:08:46-rgw-wip-rgw-d3n-filter-distro-default-smithi/7496282 2023-12-19T20:05:11.588 INFO:teuthology.orchestra.console.smithi008:Power off 2023-12-19T20:05:11.588 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-19T20:05:11.611 DEBUG:teuthology.orchestra.console.smithi008:power off output: Chassis Power Control: Down/Off 2023-12-19T20:05:11.612 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T20:05:11.627 INFO:teuthology.orchestra.console.smithi170:Power off 2023-12-19T20:05:11.627 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-19T20:05:11.636 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is on 2023-12-19T20:05:11.650 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off 2023-12-19T20:05:11.650 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T20:05:11.673 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-12-19T20:05:15.641 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T20:05:15.675 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T20:05:15.751 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is on 2023-12-19T20:05:15.784 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-12-19T20:05:19.754 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T20:05:19.785 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T20:05:19.864 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is off 2023-12-19T20:05:19.896 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off 2023-12-19T20:05:19.966 INFO:teuthology.orchestra.console.smithi008:Power off completed 2023-12-19T20:05:19.997 INFO:teuthology.orchestra.console.smithi170:Power off completed 2023-12-19T20:05:20.099 ERROR:teuthology.nuke:Could not nuke the following targets: targets: smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH81AtGnYEarGoyt8yySdPuQ2fh3SNAb5n8BGiRglCfodTFDEt+mn4LTf2kgm4W8xRqcZ3gzjZkHtAyU4bcJwgM= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGto0wgREZT7IKRt5TM8u5sT7ARHOHVhWqZg5TQ2MvUmygnqFgOG7YKxQS8XVxnmC79zL9Lh4r0Jzrrrh/lynoo= 2023-12-19T20:05:20.104 INFO:teuthology.run:Summary data: description: rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} failure_reason: machine smithi174.front.sepia.ceph.com is not locked owner: scheduled_smanjara@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=84b2a33b52284b149e4688c3fc238d6d status: fail success: false 2023-12-19T20:05:20.105 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-19T20:05:20.185 INFO:teuthology.run:FAIL