2023-12-19T01:15:21.117 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-19T01:15:21.117 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} --archive /home/teuthworker/archive/smanjara-2023-12-19_01:09:08-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi/7495647 --owner scheduled_smanjara@teuthology --verbose --name smanjara-2023-12-19_01:09:08-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi -- /home/teuthworker/archive/smanjara-2023-12-19_01:09:08-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi/7495647/orig.config.yaml 2023-12-19T01:15:21.153 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-19T01:15:21.227 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/smanjara-2023-12-19_01:09:08-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi/7495647 branch: wip-shilpa-test-wo-sync-policy description: rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} email: smanjara@redhat.com first_in_suite: false job_id: '7495647' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: smanjara-2023-12-19_01:09:08-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-shilpa-test-wo-sync-policy ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw data sync poll interval: 5 rgw md log max shards: 4 rgw meta sync poll interval: 5 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: 2d661d4af680c0d8c8034c97a83d4fed6f15d7ca ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 2d661d4af680c0d8c8034c97a83d4fed6f15d7ca rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 rgw-multisite-tests: args: - tests.py workunit: branch: wip-shilpa-test-wo-sync-policy sha1: 2d661d4af680c0d8c8034c97a83d4fed6f15d7ca owner: scheduled_smanjara@teuthology priority: 49 repo: https://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - - c1.mgr.x - c1.osd.3 - c1.osd.4 - c1.osd.5 - c1.client.1 - - c2.mon.a - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - - c2.mgr.x - c2.osd.3 - c2.osd.4 - c2.osd.5 - c2.client.1 sha1: 2d661d4af680c0d8c8034c97a83d4fed6f15d7ca sleep_before_teardown: 0 suite: rgw:multisite suite_branch: wip-shilpa-test-wo-sync-policy suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_2d661d4af680c0d8c8034c97a83d4fed6f15d7ca/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 2d661d4af680c0d8c8034c97a83d4fed6f15d7ca targets: smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNj7ZBw7+J7jmhrNmuV14R7/pGC8sjbFCzD/2UcUlkM5E/dcEzKKphQP0j+KwhNfFo2j+CCBXl07Uz2tor2PTB4= smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCN8QRIx1WeO3OkTvDNgakQiU6C2fWKN4hzl0lnNsqbWerwBj9ebTGad0SvZ8k7C18V5sYY1G/V090mXRCQryT8= smithi160.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGY+UR5DQ1tU4mLDfNqjhCwUIN91ow+eI7AfeZT/h13IDCB6inTg5DX71R6qymP1BJ4r/KSRKw2+bG4jR5abiY8= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJjpgDrlLexLbxcMQ+PvwZxPhIoI22NgVWYppU50BYWeEMhNvjoWMed5o7hcDVW40jqT9EVPuoY6Z4Elt3kSEWY= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-19_01:09:08 tube: smithi user: smanjara verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-19T01:15:21.227 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_2d661d4af680c0d8c8034c97a83d4fed6f15d7ca/qa; will attempt to use it 2023-12-19T01:15:21.228 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_2d661d4af680c0d8c8034c97a83d4fed6f15d7ca/qa/tasks 2023-12-19T01:15:21.228 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-19T01:15:21.229 INFO:teuthology.task.internal:Checking packages... 2023-12-19T01:15:21.250 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '2d661d4af680c0d8c8034c97a83d4fed6f15d7ca' 2023-12-19T01:15:21.251 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T01:15:21.251 INFO:teuthology.packaging:ref: None 2023-12-19T01:15:21.251 INFO:teuthology.packaging:tag: None 2023-12-19T01:15:21.251 INFO:teuthology.packaging:branch: wip-shilpa-test-wo-sync-policy 2023-12-19T01:15:21.251 INFO:teuthology.packaging:sha1: 2d661d4af680c0d8c8034c97a83d4fed6f15d7ca 2023-12-19T01:15:21.251 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-shilpa-test-wo-sync-policy 2023-12-19T01:15:21.412 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-203-g2d661d4a-1jammy 2023-12-19T01:15:21.413 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-19T01:15:21.421 INFO:teuthology.task.internal:no buildpackages task found 2023-12-19T01:15:21.421 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-19T01:15:21.461 INFO:teuthology.task.internal:Saving configuration 2023-12-19T01:15:21.475 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-19T01:15:21.491 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T01:15:21.516 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/smanjara-2023-12-19_01:09:08-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi/7495647', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 01:09:20.462373', 'locked_by': 'scheduled_smanjara@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDnbT4/tuiPzQ8MgmYCgIYBe7xov4vbf1pHD7HG7f842'} 2023-12-19T01:15:21.542 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/zack-2023-12-18_21:10:25-orch:cephadm:smoke-main-distro-default-smithi/7495630', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': False, 'locked_since': None, 'locked_by': None, 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIB5d//IIiXvy9CMk6u5g/LriTcib6cJVxli2RwjFUwWz'} 2023-12-19T01:15:21.542 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/check_lock.py", line 28, in check_lock assert status['locked'], \ AssertionError: machine smithi114.front.sepia.ceph.com is not locked 2023-12-19T01:15:21.606 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=24587fb51657404a8ae96b7af0439212 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/check_lock.py", line 28, in check_lock assert status['locked'], \ AssertionError: machine smithi114.front.sepia.ceph.com is not locked 2023-12-19T01:15:21.618 INFO:teuthology.nuke:Checking targets against current locks 2023-12-19T01:15:21.662 DEBUG:teuthology.nuke:shortname: smithi111 2023-12-19T01:15:21.663 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T01:15:21.686 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/smanjara-2023-12-19_01:09:08-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi/7495647', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 01:09:20.462373', 'locked_by': 'scheduled_smanjara@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDnbT4/tuiPzQ8MgmYCgIYBe7xov4vbf1pHD7HG7f842'} 2023-12-19T01:15:21.690 INFO:teuthology.nuke:Not nuking smithi114.front.sepia.ceph.com because description doesn't match: smanjara-2023-12-19_01:09:08-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi != /home/teuthworker/archive/zack-2023-12-18_21:10:25-orch:cephadm:smoke-main-distro-default-smithi/7495630 2023-12-19T01:15:21.713 INFO:teuthology.nuke:Not nuking smithi160.front.sepia.ceph.com because description doesn't match: smanjara-2023-12-19_01:09:08-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi != /home/teuthworker/archive/zack-2023-12-18_21:10:25-orch:cephadm:smoke-main-distro-default-smithi/7495620 2023-12-19T01:15:21.734 INFO:teuthology.orchestra.console.smithi111:Power off 2023-12-19T01:15:21.734 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-19T01:15:21.742 INFO:teuthology.nuke:Not nuking smithi183.front.sepia.ceph.com because description doesn't match: smanjara-2023-12-19_01:09:08-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi != None 2023-12-19T01:15:21.757 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2023-12-19T01:15:21.757 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T01:15:21.778 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-12-19T01:15:25.782 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T01:15:25.891 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-12-19T01:15:29.895 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T01:15:30.007 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2023-12-19T01:15:30.107 INFO:teuthology.orchestra.console.smithi111:Power off completed 2023-12-19T01:15:30.210 ERROR:teuthology.nuke:Could not nuke the following targets: targets: smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCN8QRIx1WeO3OkTvDNgakQiU6C2fWKN4hzl0lnNsqbWerwBj9ebTGad0SvZ8k7C18V5sYY1G/V090mXRCQryT8= smithi160.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGY+UR5DQ1tU4mLDfNqjhCwUIN91ow+eI7AfeZT/h13IDCB6inTg5DX71R6qymP1BJ4r/KSRKw2+bG4jR5abiY8= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJjpgDrlLexLbxcMQ+PvwZxPhIoI22NgVWYppU50BYWeEMhNvjoWMed5o7hcDVW40jqT9EVPuoY6Z4Elt3kSEWY= 2023-12-19T01:15:30.214 INFO:teuthology.run:Summary data: description: rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} failure_reason: machine smithi114.front.sepia.ceph.com is not locked owner: scheduled_smanjara@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=24587fb51657404a8ae96b7af0439212 status: fail success: false 2023-12-19T01:15:30.214 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-19T01:15:30.293 INFO:teuthology.run:FAIL